HashtagLegal
Software solutions for legal world.
0

Message Crawler - Version History

Version History

5.4.0

  • Teams PST: Updated PST reading API

  • Slack: Handling of deleted, edited consistent between v5 and 6 and Slack convert. Keep using 5.3.6 if you are in the middle of a project for consistency.

  • Slack: Loading large users sets

  • Bug fix in quiet time

  • Twitter: Reports count correctly on the loading screen

  • Elcomsoft: Fixed attachment parsing bug

5.3.6

  • Export DAT: Form is now resizable so could work large number of fields easier

  • Axiom XML: Improved import logic to look for attachments using "_attachmentData" field. Prior version was looking for "Attachment"

  • WhatsApp: Corrected import issue that could cause last message not be loaded.

  • Cellebrite XML: Added “Attachment List” and “Attachment Count” fields for better deduplication results.

  • Name Normalization: Import and Save CSV are now an option in this version, not just version 6

5.3.5

  • Bloomberg Email Converter: Some BCC names could be missing after conversion. Corrected email model in code to prevent this problem.

  • Assign Control Number: May generate group identifier out of sequence. Field would still work and families would be grouped together but it doesn’t look right. Corrected

  • DAT Export: If first field in table is not being exported, it will look like there is an empty first column on DAT import. All data exports fine. Corrected to make sure it looks good

  • Export with External Attachments: Export will now generate 2 cross reference files that should be move useful based on whether you using RDC or overlaying values.

5.3.3

  • EML Export: Added ability to turn on or off high performance mode. This will help users generate large EML files.

  • Slack Convert: Conversion tool will check for presence of users.json or org_users.json. This way you do not need to rename if you have corporate export.

  • Cellebrite: Support for time zone being embedded in date field. If you have dates with +05:00 be sure to upgrade to this version. This is new behavior for Cellebrite

  • Export: Eliminated dependency on compression DLL and replaced it with 3rd party DLL which seem to be more reliable. Some users reported problem with DLL error on export. Problem is computer specific and not an issue in Message Crawler.

  • Validation: Relativity has dependency on DLL from the line item above potential making my work for nothing. Will do more testing on various computers and adjust as needed.

  • Slack Convert: UNC path was not properly supported. Corrected.

  • Download Hyperlinks Tool: It would add “UNKNWN” to file extension if unable to detect. Now it will just leaves files as they are.

5.3.2

  • Teams: If you have a PST that is mix of emails and text messages, Message Crawler loads text messages and saves emails as MSGs to a separate folder for “normal” processing

  • Teams: You can specify custom MAPI fields to import from PST

  • Image Extension Detection: Small bug required both destination fields to be selected or tool won’t work. Now you can select either of the fields.

5.3.1

  • Teams HTML: Date parsing bug. First digit of year was not being read.

  • Cloud Services: Due to popular request from overseas, I have added option to select AWS location so that data doesn’t leave your country when performing translation and image identification.

  • Image Extension Detection: New tool to detect image types. This will allow you add extension to file name field so that Relativity previews attachment inline of RSMF file. Some forensic software exports some images without extension and this tool will allow users to create better quality RSMF file.

5.3.0

  • Salesforce Chatter: New data type support. Now you can load .XLSX files generated by Salesforce and convert them to RSMF. No attachment support as I don’t think you can attach files in Chatter. If you do, I would like to see some examples of such data.

  • Cellebrite: I’m seeing more and more when phone owner info is populated even if PhoneOwner set to Yes. I’m adding a check box to allow loading existing phone owner info or having it completely overwritten. I hope this help with consistency of data.

  • Team HTML: Offline option to skip downloading animated gifs and emojis. This will improve performance on computers with no internet connection available.

  • Teams HTML: Substitution of online emoji with standard once. (work in progress)

  • Teams HTML: Description of animated gif can be populated into body of the message in case attachment is not downloaded or just to improve readability of text message.

  • Potential workaround: I have reports that Elcomsoft data doesn’t always load from network drive. Copying data to C: drive solves this problem. I’m looking into what could be causing this.

5.2.5

  • Slack: Option for “Active Members Only” added to regular Slack import

  • Slack: Number of records per channel will be displayed in the log. This way you know if channel is empty from the loading screen.

  • EML Export: Changed header names from X-RSMF to X-MessageCrawer so that if you process EML files into Relativity it doesn’t think they are RSMF files.

  • EML Export: Sender information will be updated for EML files. If there are multiple messages per EML, “Multiple Senders” will be used. If only 1 message per EML, then actual person’s name will be updated.

  • RSMF Export: Validation of RSMF 2.0 files is now supported and performed automatically

  • Fixed some embarrassing spelling mistakes that we do not need to talk about. Are there more? Please let me know.

  • One of the Microsoft DLLs was missing from the application. No idea why Visual Studio stopped packaging it with application but I had to use a workaround to make sure it is present.

  • Updated multiple DLLs to newer version. I hate doing this as this opens up possibility of missing DLL during program use but updates have to be done.

  • Skype: Loaded count was 0 even if all records loaded. Corrected.

  • Facebook: Attachment path had \\ in it. Not a problem during normal use but if you are using long path notation (\\?\) Windows is not able to find attachments. Corrected so that only 1 backslash appears.

5.2.4

  • Skype: Changed date format Conversation field so that it is YYYY/MM/DD format. This way if you change groupings from conversation by day to complete conversation you do not have to resort grid.

  • TikTok: We all knew it was coming someday. Yep.. we can now get direct messages out of TikTok JSON file.

  • Slack Convert: Better reporting. A CSV file will be written to destination folder with details about progress and errors. This will help those who looking to examine errors after conversion.

5.2.3

  • Cellebrite Convert: Lots more improvement on this tool to streamline XML conversion process. Still in progress…

  • Export: When working on conversations during export step, count will be shown of current and total number of conversations

  • Ability to renumber Group Identifier fields together with DocID

  • Slack Convert: Sort by index is now numeric (before it was sorting numbers as text)

  • Teams: Error log was missing control number during import. Now control number is logged

  • Message Crawler will check for license online if locally stored license file is not found. If you have internet connection, this eliminates need to activate application and makes it easier to run multiple versions if needed.

  • Cellebrite XML: Better error trapping for corrupt XML files.

  • Activation: Better error tracking if license is not available online or if no internet connection present.

5.2.2

  • Teams HTML: Improved text extraction algorithm for complex HTML message bodies.

  • Teams HTML: Adjusted how attachments are associated with parent message. Initial logic did not work for newer exports from O365.

  • Slack: Some exports include fields with null values. This produced error converting data. Adjustments made to parsing algorithm to work around empty fields without errors

  • Slack: Long path support (keep an eye on error messages anyway as long path support is not easy)

  • Facebook Posts: “Tags” seem to be stored in new format. Updated model. Facebook is always changing so let me know if something comes up with other files.

  • Skype: Improve text extraction so that HTML characters translate to text better.

  • Export: Added a note that External Attachments are not supported for slicing in Relativity One. Slicing extracts attachments from inside RSMF file. If you export data by day, slicing is usually not required.

  • Bug Fix: Some attachments would not show up in-line and required user to click on them.

5.2.1

  • Teams: You can now link SharePoint files that contain %20 in the file path. I though it was a corruption when someone showed me this data first time but now it seems it is one of the ways Microsoft exports data. Either way, both file path formats are covered

  • Teams / Hyperlink download Tool: You can download hyperlinks that point to SharePoint server. You will need valid user name and password of cause. This is useful as Microsoft export doesn’t always export all files to SharePoint folder. It is probably some kind of bug but now you have an option to use Download Hyperlinks tool and grab those files yourself.

  • Teams HTML: Further improvements to HTML parsing. There are 2 major problems in the way HTML is formatted. I suspect it is a bug on Microsoft end and hopefully it will be fixed soon.

  • Cellebrite XML Metadata Extractor: Added ability to copy native files and generate Extracted Text for indexing. Note that this tool doesn’t parse text messages. This is just for other metadata that is not compatible with text messages.

5.2.0

  • New Tool: Teams HTML

    Microsoft is changing Teams load file format spec and instead of PST or MSG, export will be in HTML. This means full rewrite of existing tool is required. During last week or so I have planning my approach and in this version, I have first iteration of the tool. As with any tool that is this complex it will be constantly improved and adjusted as I get to see more and more sample data. At this point Message Crawler can import minimum required information to make an RMSF file but I’m sure users will want more. This is a first step and I’m looking forward to your feedback.

    Read more from Microsoft (New Teams Export Format)

  • I’m moving number of tools out of “limited testing section” and giving this version 5.2 designation.

  • Forgot to include one DLL in 5.1.9 into installer. Portable version was fine.

5.1.9

  • Further improvements to long path access algorithm.

  • More clear error reporting when it comes to missing attachments

  • New tool to split large Teams PSTs into MSGs for selective conversion (found under misc. tools menu)

  • Cellebrite XML Metadata Extractor: Added new button to write metadata to generated text files and export 2 DAT files. One with all fields and one with most common. This tool is still a work in progress.

5.1.8

  • Bug fix: Long path handling algorithm was causing problems with some short and long paths.

5.1.7

  • Bug fix: Slack convert would split conversation if multiple dates are present in same JSON file.

  • Improvements to long path handling

5.1.6

  • New Tool: iMessage Chat.DB

  • This is fairly complex database so there is a chance this tool will need to be fine tuned in the future. Let me know if you have any feedback.

5.1.5

Cellebrite XML - Metadata extractor: More object types added

5.1.4

  • New Tool: RingClone - An archiving tool for RingCentral. Export from RingClone does not have links to attachments from individual text messages. Due to this, attachments are included at the end of each conversation.

  • New Tool: Cellebrite XML - Metadata extractor. Experimental tool to pull metadata from objects that are not text messages. For example: Calls, Websites Visited, Locations, etc. This tool does NOT extract every possible field, only most relevant. Contact me if you need specific field included.

5.1.3

  • Slack Convert: Allows you to embed fields

  • Oxygen Convert Allows you to embed fields or export cross reference file

  • Cellebrite Convert: Low memory utilization conversion tool. NEW!

5.1.2

  • Improvement to start up time when no internet is available

  • Oxygen: Support for absolute path

  • Oxygen: New field “Thread By Day”

  • Slack Convert: Channel index runs sequentially when importing multiple files

  • Slack Convert: You can export grid to CSV which maybe more useful for reporting

  • Slack Convert: Additional fields for better reporting

  • Oxygen Convert: New tool for easy DAT to RSMF conversion

5.1.1

  • Optimized compression algorithm to support even larger RSMF files. (You will see temporary ZIP files in output folder during export)

  • EML export: time includes seconds information

  • Added warning for RSMF2 or external attachments that validation is not supported for those file types. Waiting on new DLL from Relativity.

5.1.0

  • Version 5 released