Recently I have been exploring the MP4 format, more specifically the ISO Base Media File Format. It appears to be quite the versatile format. Based on the general Box/Atom format. Don’t mean to go much into the format here as there are so many formats which use this structure, like Quicktime MOV, Jpeg2000, to the more recent Canon RAW CR3. I have also been digging into the DASH MP4 format, but we’ll save that for a later time.
One of the more interesting uses of MP4 lately is 360 or spherical video. They are becoming more and more popular with content creators and also used for mapping like Google street view.
A while back I picked up a Insta360 Nano S camera. It attached directly to my iPhone. With a camera on each side it could capture images and video which could later be processed to produce some interesting results.
Of course it needs to be processed first so it doesn’t look like you are peering out of your peephole. Insta360 provides software for you to process the video into a regular video or some fun creative spherical video that makes you look like you are walking on a small globe.
The formats produced by the Insta360 Nano S are plain old JPG and MP4, but uses the extensions .INSP and .INSV respectively. Neither of which are documented in PRONOM yet. But because of the nature of 360 camera’s there is a little more under the hood. If you would like to look at some samples you can find some here.
The INSP file begins like any other EXIF JPEG file, but ends with a little additional info.
The 360 cameras have some additional information from the different gyros and accelerometers, as well as GPS information. The INSP file stores much of this information after the end of the JPG format. You can also see a string of alphanumeric numbers at the end, which is consistent with most of the files I have seen. One python parser of the additional data calls it the magic number. “8db42d694ccc418790edff439fe026bf” would make a good pattern for a signature.
Mediainfo indeed sees the file as an MPEG-4 with a AVC codec, but with a invalid extension.
Complete name : VID_20210222_170428_005.insv
Format : MPEG-4
Format profile : JVT
Codec ID : avc1 (avc1/isom)
File size : 41.1 MiB
Duration : 7 s 608 ms
Overall bit rate mode : Variable
Overall bit rate : 45.4 Mb/s
Encoded date : UTC 2021-02-22 17:04:18
Tagged date : UTC 2021-02-22 17:04:18
IsTruncated : Yes
FileExtension_Invalid : braw mov mp4 m4v m4a m4b m4p m4r 3ga 3gpa 3gpp 3gp 3gpp2 3g2 k3g jpm jpx mqv ismv isma ismt f4a f4b f4v
In addition to a video and audio track, there is a text track.
Text
ID : 3
Format : Timed Text
Codec ID : text
Duration : 7 s 600 ms
Bit rate mode : Constant
Bit rate : 240 b/s
Frame rate : 10.000 FPS
Stream size : 228 Bytes (0%)
Title : Ambarella EXT
Language : English
Forced : No
Encoded date : UTC 2021-02-22 17:04:18
Tagged date : UTC 2021-02-22 17:04:18
With a little Exiftool magic, thank you Phil, we can see some of the extra data within the video file.
Serial Number : ISS2418ND7XH4H
Model : Insta360 Nano S
Firmware : v1.17.12.3_build1
Parameters : 2 947.866 946.388 964.646 0.000 0.000 90.000 942.993 2891.656 952.520 -0.682 -1.501 89.186 3840 1920 1040
Preview Image : (Binary data 578944 bytes, use -b option to extract)
Time Code : 62.155
Accelerometer : 0.0717358812689781 0.837667405605316 -0.541449248790741
Angular Velocity : -0.00380666344426572 -0.0143540045246482 0.0170918852090836
Thanks to tools like Exiftool and MediaInfo we can take a peek into some of these formats. New ways of using the existing formats and new formats entirely keep popping up making it hard to know exactly what you have. Initially I just assumed the Insta360 formats didn’t need anything extra as they just used well known format with their own extension, but I needed to look a little closer. Many other cameras are now putting additional data at the end of a standard JPG. It will be interesting to see what new ideas camera developers come up in the coming years.
GoPro has a 360 camera as well and looking at a sample .360 file, I can see it also uses an MP4 base media format, but uses two video tracks to store video from the two cameras. Might need to dig into that format soon as well.
A few years ago I became obsessed with creating 3D models from physical objects. There was an app on my iPhone called 123D Catch by AutoDesk and it allowed you to take a series of photos with your iPhone camera, then combine them to create a 3D Model. This lead me down a path to eventually take a course on Photogrammetry and develop a process for capturing objects in our Museum.
Autodesk eventually discontinued the app and built the technology into their paid products. This is when we started seeing lidar introduced with handheld devices. The first one I tried was using my XBOX360 kinect sensor with the skanect software. The quality was horrible, but was fun to learn about depth sensors and structure from motion. When the iPhone finally came out with lidar sensor it was like Apple had read my mind. I love having the ability to capture objects I find into 3D models. The quality is pretty good, not as good as taking the time to capture image sets for photogrammetry and use tools like Aigisoft metashape, but apps like Scaniverse do a fantastic job. You can check out some of the models I have captured on my Sketchfab page.
With any new technology comes new file formats, and 3D formats are definitely no exception. It seems every software developer has to come up with their own proprietary format leaving the digital preservation folks scrambling to keep up. The DPC and Archivematica published a report a couple years ago and state:
“There are many challenges in preserving 3D data. As well as the complexity of the data itself, there is a lack of interoperability between the different (often proprietary) systems that are used to create and manipulate 3D models. Relationships to other data, software and hardware also need to be captured and managed effectively.”
With my new iPhone in hand I found myself with new file format I was unfamiliar with. Universal Scene Description is a framework to exchanging 3D data between different software developed by Pixar. The relationship between Apple and Pixar goes way back so it was no surprise the Apple iPhone has support built in for this new format and I found myself capturing and sending 3D models to others with an iPhone. The USDZ format is a ZIP package format for containing a USD 3D model and is perfect for sharing and preserving.
There is no current PRONOM signatures for identifying USD formats, so I wanted to look into creating one. This is where I ran into a problem. The current PRONOM signature syntax has no way of properly identifying the USDZ format. Let me explain.
When DROID or Siegfried is used to identify a container format such as USDZ. It will first identify the format as a ZIP file, which technically it is. This triggers the software to then refer to the container signature to see if any patterns from the files internal to the ZIP match to a known format. This is done by pointing to a specific file and a hex pattern or ascii string within the file. In the case of a USDZ the internal structure may look like this:
Listing archive: scaniverse-20210928-113055.usdz
--
Path = scaniverse-20210928-113055.usdz
Type = zip
Physical Size = 5702256
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2021-09-28 11:47:36 ..... 297999 297999 scaniverse-20210928-113055.usdc
2021-09-28 11:47:36 ..... 5403849 5403849 0/texgen_0.jpg
------------------- ----- ------------ ------------ ------------------------
2021-09-28 11:47:36 5701848 5701848 2 files
In this sample file the name of the USDZ is the same name as the internal USDC file. So the name of the USDC is variable and DROID needs a static name and path to look for patterns. The USDZ specification is clear that the only required file inside a USDZ is a USD model, anything else is ancillary and is not always going to be included. Currently the only format used is USDC, but in the future may allow a simple USD or USDA format. In addition, some of the other sample files show a very nested USDC file, making identification even more difficult.
Listing archive: Scan.usdz
--
Path = Scan.usdz
Type = zip
Physical Size = 19155195
Characteristics = Minor_Extra_ERROR
Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
2021-03-09 09:22:36 ..... 19154773 19154773 /private/var/mobile/Containers/Data/Application/EFD09E66-32FB-4B08-8BED-B7E3D78FE1A8/tmp/Scan.usdc
------------------- ----- ------------ ------------ ------------------------
2021-03-09 09:22:36 19154773 19154773 1 files
The USDZ format is not the only file format which makes identification difficult through variable names and non-static patterns. An issue on GitHub has been raised to address this problem. One potential fix is to use glob patterns as suggested by the amazing Richard Lehane, creator of Siegfried. This way we could use wildcard to ignore the variable names and find any file with an extension of .USDC for example. The USDC file format has a nice 8 byte header “PXR-USDC” which is perfectly suited for identification so our container signature might look like this:
Update: I was able to get a beta version of Siegfried working with my test signature.
siegfried : 1.11.0
scandate : 2023-06-02T08:54:27-06:00
signature : default.sig
created : 2023-06-02T08:52:33-06:00
identifiers :
- name : 'pronom'
details : 'DROID_SignatureFile_V112.xml; container-signature-20230510.xml; extensions: usdz-signature-file-v1.xml; container extensions: usdz-dev1-signaturefile-20230601.xml'
---
filename : 'scaniverse-20210928-113055.usdz'
filesize : 5702256
modified : 2021-09-28T11:47:37-06:00
errors :
matches :
- ns : 'pronom'
id : 'BYUdev/1'
format : 'USDZ 3D Package'
version :
mime : 'model/vnd.usdz+zip'
class :
basis : 'extension match usdz; container name scaniverse-20210928-113055.usdc with byte match at 0, 8 (signature 1/2)'
warning :
I am still in the process of testing some beta versions of Siegfried in hopes of getting the glob matching to work, but still have more to do. Stay tuned!
Digital Preservation is all about identifying risks. This is done through a process which includes identification, validation, and metadata extraction. The more you know about the digital data you need to preserve over time, the more you can do to minimize those risks with the goal of making the data accessible over time.
Many formats are pretty straight forward, they are identifiable through a header and then have some binary bits or plain text that is readable by certain software. Others are more complicated. A common practice for more complex needs is to use a container. Word processing programs started out with plain text with maybe some formatting codes mixed in, then many moved to the Microsoft OLE container so you could have additional content embedded in a single file. Today file formats such as DOCX use a ZIP container, which houses all the text, images, formatting and anything else the format supports. Knowing what the format is and knowing what it may contain is important to preservation.
IM000959.JPG
I collect older digital cameras, specifically cameras with unique file formats, raw and otherwise. When I picked up a HP (Hewlett-Packard) point and shoot camera awhile back, I was initially unimpressed as it would only capture in a JPEG format and only 3 quality settings. While looking at a copy of the manual, I saw the camera was capable of capturing audio clips or voice memos for each photo taken. This can be handy when taking many photos and need a reminder about the context. This was not unique to HP, as many cameras could do this, normally a JPG was captured and the Audio would have the same name connecting the two. But when I recorded some audio on my little HP, placed the SD card in my computer, I couldn’t find the additional audio file. I also not the only one to ask about this.
There are many types of JPG files. Raw Streams, JPEG File Interchange Format (JFIF), and Exchangeable Image File Format (EXIF). Normally these formats have raster image data sprinkled with metadata. I have seen JPEG files embedded into other formats and containers, such as MP3, PDF, etc, but JPEG’s are not container formats. Or so I thought…..
View of HP Photosmart 433 folder in HP Photo & Imaging Gallery
Lets take a look at an image I took with my HP Photosmart 433. We’ll start with identification:
siegfried : 1.10.1
scandate : 2023-05-25T12:27:04-06:00
signature : default.sig
created : 2023-05-22T08:43:02-06:00
identifiers :
- name : 'pronom'
details : 'DROID_SignatureFile_V112.xml; container-signature-20230510.xml'
---
filename : 'GitHub/digicam_corpus/HP/Photosmart 433/IM000959.JPG'
filesize : 178922
modified : 2023-05-25T11:23:32-06:00
errors :
matches :
- ns : 'pronom'
id : 'x-fmt/391'
format : 'Exchangeable Image File Format (Compressed)'
version : '2.2'
mime : 'image/jpeg'
class : 'Image (Raster)'
basis : 'extension match jpg; byte match at [[0 16] [366 12] [178907 2]] (signature 2/2)'
warning :
IM000959.JPG was identified as x-fmt/391 which is a compressed Exchangeable Image File Format. version 2.2. Pretty straight forward. Next lets look at validation:
I removed a few lines to show important parts, but we get some similar information about the format, a JPEG with EXIF version 2.2. We also learn that HP improperly ordered their tags and put Tag 41492 out of sequence, but we can ignore that for now. Looking close at the output does not give us any indication of audio formats. There is a clue when we see the mention of a Flashpix version and additional Application Segments.
Since this is an image with EXIF data, lets also take a look at the output of Exiftool.
ExifTool Version Number : 12.62
File Name : IM000959.JPG
Directory : .
File Size : 179 kB
File Modification Date/Time : 2023:05:25 11:23:32-06:00
File Access Date/Time : 2023:05:25 11:24:42-06:00
File Inode Change Date/Time : 2023:05:25 11:24:39-06:00
File Permissions : -rwxr-xr-x
File Type : JPEG
File Type Extension : jpg
MIME Type : image/jpeg
Exif Byte Order : Little-endian (Intel, II)
Image Description : IM000959.JPG
Make : Hewlett-Packard
Camera Model Name : hp PhotoSmart 43x series
Orientation : Horizontal (normal)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Software : 1.400
Modify Date : 2021:11:16 09:04:04
Y Cb Cr Positioning : Co-sited
Copyright : Copyright 2002-2003
Exposure Time : 1/29
F Number : 4.0
ISO : 100
Exif Version : 0220
Date/Time Original : 2021:11:16 09:04:04
Create Date : 2021:11:16 09:04:04
Components Configuration : Y, Cb, Cr, -
Compressed Bits Per Pixel : 1.567552083
Shutter Speed Value : 1/30
Aperture Value : 4.0
Exposure Compensation : 0
Max Aperture Value : 4.0
Subject Distance : 1 m
Metering Mode : Average
Light Source : Unknown
Flash : Auto, Did not fire
Focal Length : 5.7 mm
Warning : [minor] Unrecognized MakerNotes
Flashpix Version : 0100
Color Space : sRGB
Exif Image Width : 640
Exif Image Height : 480
Interoperability Index : R98 - DCF basic file (sRGB)
Interoperability Version : 0100
Digital Zoom Ratio : 1
Subject Location : 0
Compression : JPEG (old-style)
Thumbnail Offset : 2046
Thumbnail Length : 7112
Code Page : Unicode UTF-16, little endian
Used Extension Numbers : 1, 31
Extension Name : Audio
Extension Class ID : 10000100-6FC0-11D0-BD01-00609719A180
Extension Persistence : Always Valid
Audio Stream : (Binary data 117820 bytes, use -b option to extract)
Image Width : 640
Image Height : 480
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:2 (2 1)
Aperture : 4.0
Image Size : 640x480
Megapixels : 0.307
Shutter Speed : 1/29
Thumbnail Image : (Binary data 7112 bytes, use -b option to extract)
Focal Length : 5.7 mm
Light Value : 8.9
Ohh, what do we have here? Exiftool mentions an audio stream. An audio stream inside the JPEG? How is this possible? The Flashpix format was originally developed by Kodak in which collaborated with HP. This was later added to the EXIF specifications. Below is an screenshot from the Exif Version 2.2 spec.
Exiftool mentioned Flashpix and additional APP2 segments. Lets take a look at the raw file in a hex editor.
Ahhh….. In one of the App2 segments we can see something familiar. A RIFF WAVE header! Lets see if we can extract the WAVE file.
exiftool -b -AudioStream IM000959.JPG > IM000959.WAV
mediainfo IM000959.WAV
General
Complete name : IM000959.WAV
Format : Wave
Format settings : WaveFormatEx
File size : 115 KiB
Duration : 10 s 681 ms
Overall bit rate mode : Constant
Overall bit rate : 88.2 kb/s
Audio
Format : ADPCM
Codec ID : 11
Codec ID/Hint : Intel
Duration : 10 s 681 ms
Bit rate mode : Constant
Bit rate : 88.2 kb/s
Channel(s) : 1 channel
Sampling rate : 22.05 kHz
Bit depth : 4 bits
Stream size : 115 KiB (100%)
MediaInfo can give us details on the embedded WAVE file, which is pretty terrible quality but is a PCM audio stream.
Embedded audio inside a raster image is not common. Most software which can render a JPEG image will most likely ignore the embedded WAVE and not even give a warning it exists. IM000959.JPG opens fine in Adobe Photoshop, but saving to a new format or making any edits will delete the WAVE file. Imagemagick also will remove the WAVE with any editing with no warning.
In order to ensure the embedded audio stream is preserved we first need to know it is there, this is where tools like exiftool can be used to extract metadata from the file and the image can be associated with having an audio stream and handled differently than any other JPEG file. More work is needed, Exiftool may mention an Audio Stream, but currently does not have the ability to pull any data from the stream.
During the 1980’s and 90’s, there was an explosion of software created for the PC and Macintosh. When it came to graphic design, Aldus, Adobe, Quark, Serif, and a few others were clearly the best. That didn’t stop other software developers in trying their hand with publishing design software. If you were on a budget, there were plenty of options to choose from. One of them, Timeworks Publisher, was very popular. It was released in 1987 for IBM PC and Atari with later releases for Apple II and Macintosh. The name was later changed to Pressworks. It was published by an interesting software company out of the UK called GST Software, also under the GSP name. They really enjoyed licensing their software.
Desktop Publishing software
TimeWorks Publisher may have been the first, but was definitely not the last. Pressworks was very popular so the software was sold and rebranded to many companies. In 2001 GST merged with eGames Europe as a new company, Greenstreet Software who continued to support the software. Some of which are:
FUJI Publisher
Global Software Publishing (in Europe) Pressworks, Power Publisher
GST Pressworks
1st Press
IMSI TurboPublisher
Media Graphics Publishers Paradise Page Express
MicroVision Vision Publisher 4
NEBS PageMagic
PersonalSoft Publications (Français)
Pushbutton Publish
Softkey Publisher DOS
Sybex Page (Deutsch)
Timeworks Publisher, Publish-it, Publisher Lite, Publish-it Lite
VCI Pro Publisher
Wizardworks CompuWorks Publisher
Instant Home Publisher
Greenstreet Publisher
Canon Publishing Suite
All the of the software listed above could open and save to the same file format with the extension .DTP with full compatibility, also used TPL for templates. Originally the DTP file format was a single proprietary binary format which had an ascii header of “DTPI” and all seemed to end with the ascii “EODF”. Later the software was enhanced to be OLE compatible and the binary format was wrapped inside. This made it work well for moving objects in and out of the software into other OLE compatible software like Word, but is confusing to format identification software as the header is the same as a Word file. I have added the two versions of the DTP format to PRONOM to help identify them better. They are fmt/1415 and fmt/1416.
Drawing Software
In addition to the popular Desktop Publishing software, there was a companion Drawing software licensed as well. It also had many titles:
BHV COLOURDRAW!
FUJI Designer
Global Software Publishing (in Europe) Designworks, Power Publisher
The Draw/Design software all used the same file format as well with the extension .ART, also with full compatibility between all the titles. The TEM extension was used for templates. Not to be confused with the AOL Image format, or Asymetrix Compel Image format, or a number of other formats using the ART extension. This format also began as a single proprietary binary format with the ascii header “GST:ART” starting at offset 16. And just like the DTP format it was later wrapped in an OLE container to be more compatible. In fact, the DTP format may have embedded Art objects! This format is not in PRONOM, so lets take a closer look.
You can see from the 1stdgn.art file here, the ascii “GST:ART” string starting at byte 16. This is consistent with all the samples I have. The first 16 bytes seem to vary in each sample and probably have to do with the size of the file and dimensions of the artwork. GST:ART is unique enough and should work well for a signature.
The ART file from a later version of Draw is in the OLE file format. This container format was designed by Microsoft as a universal container to increase compatibility among software. You can see from the hex view above the file looks very similar to the DOC format used by Word. There were many software titles which used this container format, many documented here. One of the easiest ways to look inside an OLE container is to use 7-Zip. A quick listing of the file shows it is a Type = compound and includes three files. The SummaryInformation file is common among many OLE formats and can contain some metadata, but the Contents file is what we are looking for. Examining the Contents file we find it looks identical to the earlier version of the ART format. The same “GST:ART” string starting at byte 16.
A note about the Preview.dib file. It appears to be a Device-Independent Bitmap, similar to a Bitmap file, probably for a thumbnail preview.
Writing a signature for an OLE container format is a bit more tricky. It requires a separate signature file to go along with the regular signature xml. Basically DROID is setup to “trigger” once it discovers either a “ZIP” file or “OLE” format. If it detects one of those formats it then looks into the container signature xml for additional patterns. If it finds a match then it identifies the format, if not it reports back a generic “ZIP” or “OLE” format.
As it turns out there were two different types of OLE file types, one used “Contents” for the internal file and another which used “CONTENTS”. Since the signature is case sensitive, the container signature requires two signatures both mapped to the same PUID.
These two formats were used with quite a few software titles. Hopefully these signatures cover most of them! You can find a couple samples and my signatures on my Github.
Awhile back I was asked to look at a file in our repository which had the extension OMF. It was not identified by DROID and didn’t appear to be in PRONOM. It didn’t take long to find quite a bit of information on the file format as it was used by many important software titles, or at least it used to. Exploring the details of this file format led me on quite the rabbit hole. You see, the OMFI format is based on a container format that once was heralded as the a better open choice over the Microsoft OLE container format growing in popularity.
OpenDoc
This all started with a multi-platform approach to an open document format started by Apple Computers in the early 1990’s called OpenDoc. It was originally an alliance between Apple, IBM, and Motorola. The idea was to have a framework any developer could use to develop software or components that would all work seamlessly together. Many developers were on board initially with many promised software titles being developed, but ultimately with much confusion surrounding the framework and Steve Jobs return to Apple in 1997, the project was scrapped.
Bento
The storage format to be used with OpenDoc was called Bento, in reference to the Japanese style of a compartmentalized container tray. Specifications were released in 1993.
There are four key ideas in the Bento format:
everything in the container is an object,
objects have persistent IDs,
all the metadata lives in the TOC (Table of Contents),
objects consist entirely of values, and
each value knows its own property, type, and data location.
The idea of a data model with such an organized structure was so appealing the digital preservation community there was excited to push for a Universal Preservation Format specifically for multimedia based on Bento. The idea was presented to AMIA in 1996!
Open Media Framework (OMF) Interchange
Avid Technology, a leader in audio/video editing systems, used the Bento specification to design a container format for multimedia. This allowed easy interchange of projects between many different software titles. Original specifications were published in 1994, while the 2.1 specifications released in 1997. Software titles such as Pro Tools, Cubase, Adobe Audition, Adobe Premiere, Apple Logic Pro, Apple Final Cut Pro, and many others supported the OMF format, at least for awhile. OMFI was migrated to Microsoft’s Structured Storage container format to form the core of (AAF) Advanced Authoring Format in the late 1990’s.
Identification
In order to identify an OMF file we first need to understand what is part of the OMF specifications and what is part of the Bento format. OpenDoc may not have lived very long but the Bento format held on long enough to be the structure used by a few different file formats. I am aware of the following, but there was other software being developed at the time.
Samples from each of these formats show some similar patterns. In the Bento specifications we can see:
The only version of the specifications I can find are version 1.0d5 released in 1993, but we know there was also a version 2 released later. The magic bytes are not defined in the 1.0d5 spec, but looking at the code in the Open Doc Developer Release in 1996, we can find reference to the magic bytes used in “Containr.h”.
The Bento specification also defines this header information as, “Our solution to this is to define the standard Bento format to have the label at the end of the container.” Which means this byte sequence will frequently be found at the End of File. The “CM” refers to “Container Manager” and “Hdr” refers to “Header”.
Now that we have the magic bytes for the Bento container we can look at what makes the OMF file unique from others. We can find the answer in the Bento specifications.
We know that every Bento container must have a object, so in version 1.0 of the specifications on page 65 we find.
Each object must have the property OMFI:ObjID. The value of OMFI:ObjID is required and is listed in the property description for each object.
The OMFI:ObjID can also be found in version 2.0 of the specification, but in addition it defines:
The OMFI:ObjID property has been renamed the OMFI:OOBJ:ObjClass property, which eliminates the concept of generic properties and makes the class model easier to understand. The name ObjClass is more descriptive because the property identifies the class of the object rather than containing an ID number for the object.
Since both are required it seems appropriate to use those strings for identification in a PRONOM signature. You can check out the proposed signature and samples on my GitHub page.
There is so much history wrapped up in these formats and the potential they had to change how we preserve files in our archives. Luckily we have the Internet Archive WayBack machine to help us discover or remember ideas that once existed, some which may find their way back to inspire future file formats.
