6.7.0 - April 20, 2024
Updated to how dates are stored in RSMF for RelativityOne users. Added Z at the end of the date as per new specs so that dates do not get adjusted with local time zone settings on users’ computers. Server users are not affected.
New Tool: Extract ZIPs. This tool will extract ZIP files and will allow you to exclude them from export. This is an experimental tool. Please share feedback.
New Tool: Link native files. If you have folder with native files you can easily overlay it just by pointing to a folder.
New Tool: Create searches. More ways to query entire database to make searches for export.
WhatsApp: Improvements in iPhone parsing so that [ ] are handled better when they show up in text message body.
Download Hyperlinks: Can save files in repository to organize files better.
Export RSFM: Added option to split RSMF files by size.
Import/Convert Slack: Naming parsing is more consistent when working with "broken" exports. Broken = member names list for channel is not complete, users are not in users.json list, etc. Good/complete exports will work the same as before.
Updated lots of backend DLLs: RSMF Validator, PST parser, thumbnail generator, etc.
Teams HTML v2: "Reactions" were loaded to new field Reactions and not existing "Message Reactions" field. This was not causing issues but was inconsistent. If you have saved presets, please adjust them.
Axiom: Added another field to list of possible From.
6.6.1 - March 9, 2024
New Tool "Create Searches": If you have super large import that you cannot load to grid due to size, you can now create multiple search to export it out in smaller sections.
Axiom XML: Major updates to how conversation field is generated. Different types of exports from Axiom require different parsing logic and different fields to generate conversations. Using very diverse export, I was able to finetune the logic to where I feel it works very well.
Main UI: Added button to sort documents with 1 click.
Teams HTML: Sometimes images would be download as GZ files. Added function to unzip them to their native PNG format.
Teams HTML Convert: Added option to specify names delimiters for Email_to column. Seems like different people have different delimiters and no good indication why.
Teams HTML Convert v2: More metadata fields will be used creating even better RSMF file. (reactions, deleted, edited, friendly conversation)
Slack: Added button to save log. Now you don't have to copy and paste it.
Oxygen XML: Checking for field Created that will be used if no Timestamp field present.
Export EML: Local time zone will not show up in the header. It will be reset to 0.
Export RSMF: Relativity will now be able to deduplicate RSMF files. Changed how Participant IDs are generated. Relativity's documentation states that Name and Email are used for deduplication. This is not correct and support staff member stated that IDs are being taking into account as well. Due to this discrepancy, deduplication on RSMF files was not effective. New way of generating participants IDs should resolve this problem.
RSMF Export: Added more items to drop for Messaging Platform.
Cellebrite: Bug fix. Date deleted may get offset to wrong time zone when using SQLite database.
Slack Convert: If channel name is over 50 characters, it will be hashed for purposes of document prefix. Long channel names cause problems in Relativity Server 2022 (2023 is fine)
Database Creation: Added an artificial delay to allow Azure to create database before tables will be created.
Bloomberg IM: When user joins or leaves, message body will populated with appropriate text. Nice for EML/PDF export. Also, I removed word Day from file name as it was confusing to many. Just using _0001, _0002 suffix
6.6.0 - January 11, 2024
Teams HTML v2: Further improvements. Proper parsing for "user left" conversation. Better handling for anonymous entities.
Teams HTML v2 Convert: Added convert option that uses new code from Teams HTML v2.
Export RSFM: Modified On and Modified By are automatically mapped
Export RSFM: Squeezed elements on export screen so that Conversation Identifier and Freendly Name drop downs fit better.
Slack Convert: Minor bug fix, if you do not select JSON file and cancel, application generates an error. Fixed.
Slack and Slack Convert: If file "canvas_in_the_conversation.json" is present it will cause Message Crawler to crash and not fully import a channel. Add code to ignore them.
Oxygen XML: You can specify not just prefix but starting number as well. Same as most other tools
Oxygen XML: Better error reporting for missing attachments. You will get error if attachment record is present but has no file path or if file path doesn't point to an actual file.
Cellebrite XML: Added option "Treat Name and Identifier as separate entries when checking if blank." This is for advanced users who want to tinker with their data. This maybe powerfull tool for some and cause problems for others. Test before using on real project.
Telegram: Better progress reporting so it doesn't look like application is frozen
Telegram: If sender name is blank, ID will be used. Few other similar edge cases accounted for.
Telegram: Improved Conversation (will use ID if name is blank) and added Conversation Complete
Backend: Updated Microsoft.Data.SqlClient which has published vulnerability. This is not a big deal in this case but IT panics so... (https://github.com/advisories/GHSA-98g6-xh36-x2p7)
Names Normalization and Names Search: Added column showing how many times name appears.
Axiom: Added more fields to look for To field. (Axiom can use 6-7 fields with different names)
Facebook: Changed how attachments are found. No longer you need specify 2 paths. Message Crawler will scan starting path and will use file names to match files to JSON. This increased loading speed and easy of use significantly.
Facebook: Will only scan for message_*.json to avoid errors with other random json files that can be present in the folder.
Facebook: FileSource path would sometimes have \\ instead of \ on occasion. It did not cause any problems but it was fixed anyway for consistency.
Bloomberg IM Import: Improved UI, error logging and lots of little fixes here and there.
Bloomberg IM Convert: First beta version of Bloomberg Instant to RSMF conversion tool.
Main Screen: When switching projects, grid displays old data. Grid will be cleared when new project is opened.
6.5.3 - January 7, 2024
Import Slack: Sometimes HTML characters would be present in body of the message. Added code to decode them.
Import Teams HTML v2: I'm seeing new format for Teams HTML export. It is compatible with old import tool but has more info such as reactions, replies and more metadata. This version introduces beta support for new Teams HTML format. How do you know if you have new format? If you look at origianl HTML, reactions will be shown as text and not an embedded picture.
Export: Updated RSMF validator to most recent DLL.
Export: EML export will populate full path in cross reference
6.5.2 - December10th, 2023
Import Oxygen XML: In some cases Oxygen can export date twice for one record. This would load ok to SQLite but not SQL. SQL loading would stop part way. Added handling this very rare case.
PDF Export: Slightly better handling of PNG files that fail to embed in PDF
PDF Export: Added line break between attachments, in case multiple once are present in same record.
Export EML: Added ability to write fields to body of EML so they are searchable.
Import RSMF: Support for super huge RSMF files. Tested with 12gig RSMF and it can be loaded to Message Crawler.
Signed EXE: Message Crawler is now a signed application, meaning no more "This application is from unknown publisher" message.
6.5.1 - November 19th, 2023
Export: Bug fix in "Copy Files" and "Generate Native". File path was not being overlayed into the database
Export PDF: Option to exclude attachment warning.
Import Twitter: Additional fields added with reply to info and deleted indicator.
Bug Fix: Custom fields named with some special characters would not be updated in the database. UI would be updated making user think data is saved. Requerying data would show blank fields.
Import Teams HTML: Extracted emojis will be shown only once per message, no duplicates.
Export: Added fields with First Message Date and Last Message Date to cross reference. This is helpful when you are creating PDFs and need to overlay metadata.
Import/Search Edit Details: Window can be resized so you could see and edit query.
6.5.0 - October 23th, 2023
Import: New type: XRY
Export: Added Exported Message Count to cross reference
Export: Added Start and End dates to exported DAT
Export: Added export for grid directly to Excel.
Cellebrite New Version: New fields Deleted Date and Deleted Reason
Cellebrite New Version: Instant Message section may produce an error during import. This is due to missing fields in new XML. Adjusted code to handle missing values and not error out.
Cellebrite: New field - Label. This field may contain idication if message was edited. Practical use of this field is to be determined. Right now it is just being captured.
Slack Bug: Date filter in the import screen was not actually doing anything. Removed until I figure it out.
PDF Export: Emojis are converted from text form to real once when possible.
HTML Export: Reactions are supported for newer HTML format.
HTML Export: Emojis are converted from text form to real once when possible.
SQL Credentials: Changed how credentials are being checked potentialy avoiding odd error.
WhatsApp: You can now set starting number and automatically run deduplication. Kind of forgot about this tool but now it is on par with others. Figuring out date format is still a pain but not much I can do about it.
Facebook/IG: Share information can be added to body of the message.
Telegram: Added ability to attempt to lookup phone number of the user and append after their name.
6.4.0 - July 29th, 2023
Export: New export type - Universal Export. This will allow you to export multiple formats (RSMF/HTML/PDF) at the same time. Standard fields will be used.
Export: PDF - Images are slightly large and fixed issue with images shifting.
Export: Preview button will show you HTML rendering of the output files. This way you know if you got file mappings correct before exporting.
Export: HTML2 export option. You can now export HTML files that are used to generate PDFs. Look for "Use new HTML format" in the HTML export screen.
Export: HTML2 export will generate cross ref file when copying attachments to folder. You should still use some kind of processing tool ingest.
Export: PDF export will be single threaded by default as it is too CPU intensive and works well only on most recent physical computers.
Export: PDF export will generate cross ref file when copying attachments to folder. You should still use some kind of processing tool ingest.
Axiom: Major updates to import module. A lot of fields will not be populated and extra fields prefixed with Axiom_. QCing data and sorting still required.
Oxygen XML: Update to import logic for XML files without From or To fields.
Generate Thumbnails: Will updated database every 10 files, just in case application crashes.
Generate Thumbnails: Improved performance for Office type files.
Download Hyperlinks: Will updated database every 10 files, just in case application crashes.
SQL: Document table will have compression turned on to save space.
Name Normalization: Shows progress while running
Name Normalization: Fixed an error that would come up if Original Name is not found. While this should not happen, some people export data into Excel and then alter Orig Name. This is bad and will prevent tool from working properly. Added code so that tool would not crash.
6.3.1 - June 27nd, 2023
Slack and Slack Convert Bug: Memory is not being cleared correctly between imports. If you close and open import screen between JSON files, data will be loaded correctly. If you select new JSON after loading prior one and do not close import screen, old channels will stay in memory and produce incorrect results. You can spot this issue where number channels on the list is different from number in the toolbar below. If you have NOT closed import/convert screens between steps, I recommend reconverting that data that was processed in version 6.3.0
6.3.0 - June 22nd, 2023
Fields Change #1: Facebook reactions now saved to regular field "Message Reactions" and not custom field.
Fields Change #2: Custom fields added to RSMF header are now prefixed with X-RSMF to make them compatible with new feature released in RelativityOne that allows you to map custom RSMF fields.
Axiom XML: Added support multiple "sections" in XML file. Each section can have different fields which adds even more complexity to working with this data.
Import: MBOX Import for 3 types of MBOX files: Normal, Google Vault, Android text message sync to Gmail. (This is first version for testing as parsing this data is really hard and it may not be perfect)
Convert Oxygen: Added option to use custom prefix for file naming.
Generate Thumbnail for Video: Generated "unable to delete temp folder" error but was working fine. Suppressed the error.
Slack: You can now scan folder for JSON files. If any of the main 4 files found, they will be loaded (channels, groups, dms, mpims)
Slack: New field "Slack JSON Source JSON". It shows what JSON file was used to load other JSON fiels. (channels, groups, dms, mpims)
Slack Convert: Convert Slack to PDFs directly.
Generate Thumbnail: You can generate thumbnails for office type files and PDFs. Give it a try. (no official supported file type list)
Export: "Large Attachments" folder only generated when needed.
Belkasoft: Improve import algorithm to account for more fields.
Bug Fix: In Advanced Duplication tool, Saved Query read from import set instead of query. Wrong query was displayed and used.
Bug Fix: Database name was no being updated in the title bar when switching between SQL databases.
Bug Fix: Slack Convert EML conversion with sub threads may not get written to folder correctly. This issue can be identified by looking at expected RSMF/EML count column and comparing to actual RSMF files in the folder. It is fixed.
Bug Fix: Slack Convert cross ref file may not have correct path in Export FIle Name column. Full path is correct and can be used in cross reference.
Bug Fix: Slack URL may not be generated in correct path.
Bug Fix: Slack possible error when selecting multiple JSON files resolved.
Bug Fix: Axiom XML, If XML contains field that already exists as system field in Message Crawler, error would be generated.
Bug Fix: Advanced DAT Import may produce error "Field already exist"
6.2.0 - May 13th, 2023
Export: New export type - PDF
Cellebrite: New field Parent Identifier (for supported data)
Cellebrite: Checks for field URL in the attachment object. If present, it will be appended to body of the message.
Cellebrite: Facebook data could generate an error during import due to different fields. Adjusted code to account for that.
Telegram: New field Parent Identifier, Telegram Sequence Number, Telegram ReplyTo Number (sequence numbers are not unique across conversations)
Export: If field Parent Identifier populated, it will be exported to RSMF enabling indent of messages to better indicate when someone replies to a specific message
New Tool: Generate direction. This tool can be used to generate field Direction that can be used in RSMF or PDF to indicate incoming or outgoing message
Import Dir list: Bug prevented import from working if hashing is turned off.
Generate Thumbnails: More file types are supported.
Teams PST: Identified situation where data may not be sorted after import. Made sure it will be sorted automatically however sorting is done on folder level. Additional sort maybe required.
Teams HTML: Field Email_to was being automatically created which was confusing. Data is mapped to Message To, field so Email_to is not required.
6.1.1 - April 23rd, 2023
Oxygen DAT Convert: Fixed field naming bug that prevented this tool from working
Cellebrite XML: If you do not select XML, you will get message Import Complete. Removed this message.
Cellebrite XML: New field - Direction. Phone owner = outgoing, rest = incoming.
Grid: Sort order would not stick at times if using rows and column headers for sorting and opening a tool. Big sort button was working fine which is what most people use.
File Identification: Some drop downs were pre populated with wrong values.
Video Thumbnail Generation: Improved extraction algorithm and made playback smother.
6.1.0 - April 2nd, 2023
First production release of version 6