RealVideo

For #WDPD24 and PRONOM Hackathon week this year, I want to find some older formats listed which did not have a signature. There is a list to choose from, but I wanted to find something I hadn’t worked on before. I came across two entries for Real Video:

PUIDNameExtension
fmt/204RealVideo Cliprv
x-fmt/277Real Videorv

I was familiar with Real Media and Real Audio, but had yet to come across any RealVideo with the RV extension. I thought it would be easy to find some references and samples, but that was not the case. I assume PRONOM originally added these based on MIME types available.

Real or RealNetworks is/was an Internet media company who jumped on the rapidly growing World Wide Web in 1995 to become a leader in Internet Media Delivery. Their initial offerings mainly focused on audio streaming and they accomplished all of this by providing free players and web browser extensions to make it easy to serve up a website with streaming media everyone could enjoy. Later adding video streaming optimized for the slower dialup and connections of the day. They used codecs based on common technology like H.263 and H.264, but used then to make their own proprietary codecs identified through FourCC codes, RV10-RV60.

So thought it would be easy to find a reference to the RV extension, I quickly discovered it wasn’t. Looking at the Wikipedia page on RealVideo, I found no reference to the RV extension. RV is an abbreviation for RealVideo, right? Well, I ended up finding a reference in the RealAudio page under file extensions. Ok, First clue to the existence of the RV extension. The page references RV as being used for video only files and was used by the flagship encoder (RealProducer).

RealProducer was the tool for creating the streaming audio and video formats that could then be used for your website or streaming platform. The RealProducer software came in a Basic version, which was free, and the Plus or Pro version, which was not free and provided more options. The first version of RealProducer to make video files was version 4. I was able to find a copy of the encoder and installed it under a Windows 95 emulator. To my surprise it only saved to the RealMedia RM file format. This format is well known and identified with PRONOM as x-fmt/190 also documented at the LoC.

This was the same with RealProducer 5, 7, 8, 9, and 10 that I was able to try. All made no mention of the RV extension. I was starting to feel this format didn’t exist or that some decided to use the RV extension on their own. Searches on Google yielded a couple results, mostly from users who had found a few files on their older discs and wanted to migrate them to something newer. I was able to find one example, one user shared, but it had the same header as the RealMedia format. The clue was in the file.

hexdump -C ambush_abb.rv
00000000  2e 52 4d 46 00 00 00 12  00 01 00 00 00 00 00 00  |.RMF............|
00000010  00 07 50 52 4f 50 00 00  00 32 00 00 00 03 6e e8  |..PROP...2....n.|
00000020  00 03 6e e8 00 00 03 e0  00 00 01 b3 00 00 6a 6f  |..n...........jo|
00000030  00 06 80 fa 00 00 08 b5  00 ba 41 73 00 00 03 55  |..........As...U|
00000040  00 03 00 09 43 4f 4e 54  00 00 00 40 00 00 00 00  |....CONT...@....|
00000050  00 00 00 08 28 43 29 20  32 30 30 35 00 26 00 00  |....(C) 2005.&..|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000270  00 09 61 75 64 69 6f 4d  6f 64 65 00 00 00 02 00  |..audioMode.....|
00000280  06 76 6f 69 63 65 00 00  00 00 2d 00 00 0d 43 72  |.voice....-...Cr|
00000290  65 61 74 69 6f 6e 20 44  61 74 65 00 00 00 02 00  |eation Date.....|
000002a0  13 39 2f 32 30 2f 32 30  30 36 20 31 34 3a 30 37  |.9/20/2006 14:07|
000002b0  3a 30 38 00 00 00 00 53  00 00 0c 47 65 6e 65 72  |:08....S...Gener|
000002c0  61 74 65 64 20 42 79 00  00 00 02 00 3a 52 65 61  |ated By.....:Rea|
000002d0  6c 50 72 6f 64 75 63 65  72 28 52 29 20 42 61 73  |lProducer(R) Bas|
000002e0  69 63 20 31 31 2e 30 20  66 6f 72 20 57 69 6e 64  |ic 11.0 for Wind|
000002f0  6f 77 73 2c 20 42 75 69  6c 64 20 31 31 2e 30 2e  |ows, Build 11.0.|
00000300  30 2e 32 30 30 39 00 00  00 00 31 00 00 11 4d 6f  |0.2009....1...Mo|
00000310  64 69 66 69 63 61 74 69  6f 6e 20 44 61 74 65 00  |dification Date.|
00000320  00 00 02 00 13 39 2f 32  30 2f 32 30 30 36 20 31  |.....9/20/2006 1|
00000330  34 3a 30 37 3a 30 38 00  00 00 00 1d 00 00 09 76  |4:07:08........v|
00000340  69 64 65 6f 4d 6f 64 65  00 00 00 02 00 07 6e 6f  |ideoMode......no|
00000350  72 6d 61 6c 00 44 41 54  41 00 ba 3e 1e 00 00 00  |rmal.DATA..>....|

RealProducer Basic 11 for Windows. The Wikipedia article did hint at this by saying “the latest version of RealProducer reverted to using .ra for audio only files and began using .rv for video files with or without audio.” Why would they use the RM extension for so long, then revert to a different extension with a later version? I found more in the User Manual for version 11.

• .rv – RealVideo
RealProducer uses the .rv file extension if the input is video-only or video-with-audio. You can also select the .rm file extension for video content.
Tip: Using the .rv file extension helps search engines identify the file as a RealVideo clip.

• .rm – RealAudio or RealVideo
RealProducer chooses the .rm file extension if it cannot determine the content of the input clip. You can use .rm file extension for any RealAudio or RealVideo clip, except for variable bit-rate clips.

Ok, so a few things to learn from this. One is the RV extension was used as the default for version 11 as they wanted search engines to identify them as a RealVideo clip. Second thing we learned is there is no difference between the two placeholders in PRONOM, one being a RealVideo file and the other being a RealVideo Clip. We don’t need both.

Now, is there any difference between an RV and RM file?

hexdump -C Producer11-01.rv | head
00000000 2e 52 4d 46 00 00 00 12 00 01 00 00 00 00 00 00 |.RMF............|
00000010 00 07 50 52 4f 50 00 00 00 32 00 00 00 03 6e e8 |..PROP...2....n.|
00000020 00 03 6e e8 00 00 03 e0 00 00 01 c7 00 00 01 66 |..n............f|
00000030 00 00 1b 57 00 00 07 41 00 02 91 0a 00 00 03 5e |...W...A.......^|
00000040 00 03 00 09 43 4f 4e 54 00 00 00 40 00 00 00 00 |....CONT...@....|
00000050 00 00 00 08 28 43 29 20 32 30 30 35 00 26 00 00 |....(C) 2005.&..|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000080 00 00 00 00 4d 44 50 52 00 00 00 70 00 00 00 00 |....MDPR...p....|
00000090 00 02 c2 a4 00 02 c2 a4 00 00 03 e0 00 00 01 9f |................|

hexdump -C Producer11-01.rm | head
00000000 2e 52 4d 46 00 00 00 12 00 01 00 00 00 00 00 00 |.RMF............|
00000010 00 07 50 52 4f 50 00 00 00 32 00 00 00 03 6e e8 |..PROP...2....n.|
00000020 00 03 6e e8 00 00 03 e0 00 00 01 a4 00 00 01 64 |..n............d|
00000030 00 00 1b 57 00 00 05 a4 00 02 5c 35 00 00 03 5e |...W......\5...^|
00000040 00 03 00 09 43 4f 4e 54 00 00 00 40 00 00 00 00 |....CONT...@....|
00000050 00 00 00 08 28 43 29 20 32 30 30 35 00 26 00 00 |....(C) 2005.&..|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000080 00 00 00 00 4d 44 50 52 00 00 00 70 00 00 00 00 |....MDPR...p....|
00000090 00 02 c2 a4 00 02 c2 a4 00 00 03 e0 00 00 01 a4 |................|

They both look very similar to me. Aside from a few bytes, they are practically identical. Lets see what MediaInfo has to say.

mediainfo Producer11-01.rv
General
Complete name : Producer11-01.rv
Format : RealMedia
File size : 164 KiB
Duration : 6 s 999 ms
Overall bit rate : 225 kb/s
Frame rate : 24.000 FPS
Copyright : (C) 2005
FileExtension_Invalid : rm rmvb ra

Video
ID : 0
Format : RealVideo 4
Codec ID : RV40
Codec ID/Info : Based on AVC (H.264), Real Player 9
Duration : 6 s 999 ms
Bit rate : 181 kb/s
Width : 640 pixels
Height : 424 pixels
Display aspect ratio : 3:2
Frame rate : 24.000 FPS
Bits/(Pixel*Frame) : 0.028
Stream size : 155 KiB (94%)

Audio
ID : 1
Format : Cooker
Codec ID : cook
Codec ID/Info : Based on G.722.1, Real Player 6
Duration : 7 s 429 ms
Bit rate : 44.1 kb/s
Channel(s) : 2 channels
Sampling rate : 44.1 kHz
Bit depth : 16 bits
Stream size : 40.0 KiB (24%)

mediainfo Producer11-01.rm
General
Complete name : Producer11-01.rm
Format : RealMedia
File size : 151 KiB
Duration : 6 s 999 ms
Overall bit rate : 225 kb/s
Frame rate : 24.000 FPS
Copyright : (C) 2005

Video
ID : 0
Format : RealVideo 4
Codec ID : RV40
Codec ID/Info : Based on AVC (H.264), Real Player 9
Duration : 6 s 999 ms
Bit rate : 181 kb/s
Width : 640 pixels
Height : 424 pixels
Display aspect ratio : 3:2
Frame rate : 24.000 FPS
Bits/(Pixel*Frame) : 0.028
Stream size : 155 KiB

Audio
ID : 1
Format : Cooker
Codec ID : cook
Codec ID/Info : Based on G.722.1, Real Player 6
Bit rate : 44.1 kb/s
Channel(s) : 2 channels
Sampling rate : 44.1 kHz
Bit depth : 16 bits

Other than the RV file having a invalid file extension, they both identify as a RealMedia file and have identical properties. So it seems the RV file is really no different than the RM file. I think the best course of action for PRONOM is to deprecate these two RV PUID’s and just ad RV as an acceptable extension for the RealMedia format.

To add to the evidence, here is the output from ffprobe:

Input #0, rm, from 'Producer11-01.rm':
Metadata:
copyright : (C) 2005
comment :
ASMRuleBook : #($Bandwidth >= 0),Stream1Bandwidth = 44100, Stream0Bandwidth = 180900;
Audiences : 256k DSL or Cable;
audioMode : music
Creation Date : 11/12/2024 20:28:55
Generated By : RealProducer(R) Plus 11.1 for Windows, Build 11.1.0.2676
Modification Date: 11/12/2024 20:28:55
videoMode : normal
Duration: 00:00:07.00, start: 0.000000, bitrate: 176 kb/s
Stream #0:0: Video: rv40 (RV40 / 0x30345652), yuv420p, 640x424, 180 kb/s, 24 fps, 24 tbr, 1k tbn
Stream #0:1: Audio: cook (cook / 0x6B6F6F63), 44100 Hz, stereo, fltp, 44 kb/s

Input #0, rm, from 'Producer11-01.rv':
Metadata:
copyright : (C) 2005
comment :
ASMRuleBook : #($Bandwidth >= 0),Stream1Bandwidth = 44100, Stream0Bandwidth = 180900;
Audiences : 256k DSL or Cable;
audioMode : music
Creation Date : 11/12/2024 20:28:16
Generated By : RealProducer(R) Plus 11.1 for Windows, Build 11.1.0.2676
Modification Date: 11/12/2024 20:28:16
videoMode : normal
Duration: 00:00:07.43, start: 0.000000, bitrate: 181 kb/s
Stream #0:0: Video: rv40 (RV40 / 0x30345652), yuv420p, 640x424, 180 kb/s, 24 fps, 24 tbr, 1k tbn
Stream #0:1: Audio: cook (cook / 0x6B6F6F63), 44100 Hz, stereo, fltp, 44 kb/s

But wait, there are a couple formats we could add which are related to RealProducer. RealProducer used a few other formats to manage projects and other metadata for streaming. They include:

  • .RP RealPix Image
  • .RT RealText
  • .RPAD RealProducer Audience File
  • .RPJF RealProducer Job File
  • .RPSD RealProducer Server Destination
  • .RMHD RealMediaHD file
  • .RAM Playlist
  • .RPM Embedded RAM
File TypeExtensionMIME Type
Ram.ramaudio/x-pn-realaudio
Embedded Ram.rpmaudio/x-pn-realaudio-plugin
SMIL.smil and .smiapplication/smil
RealAudio.raaudio/x-pn-realaudio
RealVideo.rmapplication/x-pn-realmedia
Flash.swfapplication/x-shockwave-flash
RealPix.rpimage/vnd.rn-realpix
RealText.rttext/vnd.rn-realtext
https://web.archive.org/web/20120513203726/http://service.real.com/help/library/guides/production8/htmfiles/server.htm

Don’t get excited, the RealPix Image format really isn’t an image, it is simply an XML file with all the details of an image or group of images. Pretty boring. It was however a big thing in the day, even got a full guide written up for the process. “All information in the file occurs between an opening <imfl> tag and a closing </imfl> tag. This is the only tag that uses an end tag.” This format was the topic of discussion as malicious code could be in the RP file and executed just by having someone load your webpage. IMFL is obviously an acronym, but none of the documents I could find tells me what it stands for, so I did what everyone does now, I asked ChatGPT.

The RealPix format by RealNetworks, which was used for interactive multimedia content, indeed utilized IMFL as its tagged format. IMFL stands for “Interleaved Media File Language.” This markup was particularly designed to handle multimedia presentations, allowing the synchronization of images, audio, and video in a slideshow-style format. It used XML-like syntax where elements like <imfl>, <head>, and <fadein/> defined media objects, transitions, and their timing. Key components included attributes for positioning, color, and animation effects, making RealPix a flexible format for creating multimedia sequences compatible with RealPlayer.

For technical details, the RealPix format closely resembles SMIL (Synchronized Multimedia Integration Language) and supports strict tag closure and case sensitivity. This means all tags and attribute names must be lowercase, and attributes must be in double quotes, as seen in SMIL and RealSystem G2 markup, RealNetworks’ broader multimedia framework.

When I asked for a source, it could not give me one. So not sure if it is the correct answer, but it seems to fit. Here are some samples of RP, RT and SMIL files.

For RealText with the RT extension, we find a similar tagged text. This format is used to provide text presentations to go along with Images, Audio, or Video. The tagged text then describes when and how the text is displayed. This is all done in a player window, therefore the root tag of these RT documents starts and ends with <window>. I guess these could be considered a subtitle format for streaming media.

The SMIL files is interesting, it is known standard, but in many cases, does not have an XML declaration, therefore not identified by current PRONOM. They are used to link everything together. I might suggest a variant of the SMIL format to not have the XML declaration to identify these formats correctly.

<smil>
<body>
<par>
<textstream src=”rtsp://realserver.company.com/mary.rt”/>
<video src=”rtsp://realserver.company.com/mary.rm”/>
</par>
</body>
</smil>

The .RPAD RealProducer Audience File, .RPJF RealProducer Job File, .RPSD RealProducer Server Destination are all XML files for managing some of the configuration found in the RealProducer software.

cat 56k\ Dial-up.rpad
<?xml version="1.0" encoding="UTF-8"?>
<audience xmlns="http://ns.real.com/tools/audience.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ns.real.com/tools/audience.2.0 http://ns.real.com/tools/audience.2.0.xsd">
<avgBitrate type="uint">34000</avgBitrate>
<maxBitrate type="uint">68000</maxBitrate>
<streams>

cat RealProducer11-01.rpjf
<?xml version="1.0" encoding="UTF-8"?>
<job xmlns="http://ns.real.com/tools/job.2.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://ns.real.com/tools/job.2.0 http://ns.real.com/tools/job.2.0.xsd">
<enableTwoPass type="bool">true</enableTwoPass>
<clipInfo>

cat Multicast\ Push\ Server.rpsd
<?xml version="1.0" encoding="UTF-8"?>
<destination xsi:type="pushServer" xmlns="http://ns.real.com/tools/server.2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ns.real.com/tools/server.2.0 http://ns.real.com/tools/server.2.0.xsd">
<pluginName type="string">rn-server-rbs</pluginName>

Those three formats should be easy enough, especially if we look for Namespace urls.

The RAM and RPM formats are simply text files with a URL. You can find some samples here and here.

An RM and RV file are the same format as the RMVB file but just with a variable bitrate. Later on a new format was used to improve the quality of video. This format has the extension RMHD, referring to RealMedia HD. Let’s take a look.

hexdump -C DSC_0009.rmhd | head
00000000 2e 52 4d 50 00 00 00 12 00 01 00 00 00 00 00 00 |.RMP............|
00000010 00 07 50 52 4f 50 00 00 00 36 00 02 00 04 f7 33 |..PROP...6.....3|
00000020 00 04 f7 33 00 00 11 bd 00 00 02 5d 00 00 01 d2 |...3.......]....|
00000030 00 00 1b 2e 00 00 00 00 00 00 00 00 00 04 65 68 |..............eh|
00000040 00 00 01 6f 00 02 00 03 43 4f 4e 54 00 00 00 12 |...o....CONT....|
00000050 00 00 00 00 00 00 00 00 00 00 4d 44 50 52 00 00 |..........MDPR..|
00000060 00 76 00 00 00 00 00 03 24 64 00 03 24 64 00 00 |.v......$d..$d..|
00000070 11 bd 00 00 04 2a 00 00 00 00 00 00 00 00 00 00 |.....*..........|
00000080 1b 2e 0c 56 69 64 65 6f 20 53 74 72 65 61 6d 14 |...Video Stream.|
00000090 76 69 64 65 6f 2f 78 2d 70 6e 2d 72 65 61 6c 76 |video/x-pn-realv|

The format looks very similar, but has the magic header of .RMP instead of .RMF. MediaInfo and FFProbe are unaware of the format. The software mentions a RV11 codec which is confusing as the codecs went from RV10-RV60.

Phew, that was a lot considering the two formats I tried to research came up the same as an existing format. There are probably others I have missed. I did see a reference to an RMX format which seems to be an encrypted RM file. The header is the same so it will identify as a RealMedia file, but with the wrong extension. Let me know if you come across any. I have some samples of the formats mentioned here, plus a proposal of new signatures on my Github repository.

PAR

Some file formats have a unique extension. Some formats use three character extensions which are well known, so its not common for them to be used with other software. Take the extension PDF for example, pretty sure no one else will use it as it is so well known. Other extensions often get reused by a few different software titles. There are plenty of titles which use the DOC extension.

Part of defining a file format I come across is also defining other formats which use the same extension or the same basic patterns within the format. I want the format I am researching to be identified correctly, but I also don’t want other formats to falsely identify as them either.

When using the DROID tool, if a file can’t be identified using a signature, the tool will then look to see if the extension matches any formats within the PRONOM registry, if it finds one, it will identify as that format with the identification method as “Extension”. This can be confusing and dangerous.

The topic of a format came up recently in reference to the extension PAR. Lets take a look at what we know about files with the extension PAR. Using the handy tool at digipres.org, we can see there are many formats using the PAR extension.

Apparently many people like to use the extension with their software. One might think their files with the PAR extension have to be in this list, and they would be wrong in that assumption. The PRONOM registry has no records of any format using the PAR extension. Hopefully we can add a few to help with proper identification instead of using the extension only.

A PArchive or Parity Volume Set is a group of file formats used in error correction and data integrity. Only the first version used the PAR extension, it is now obsolete with version 2 being the last stable version.

hexdump -C archive.par | head
00000000 50 41 52 00 00 00 00 00 00 00 01 00 00 09 00 02 |PAR.............|
00000010 8f d0 ce 2e 21 db 3b e5 41 d5 18 be d3 0e 52 f0 |....!.;.A.....R.|
00000020 de b6 b3 9f 53 09 ff ba 16 6b ca d2 48 a6 ca 45 |....S....k..H..E|
00000030 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000040 60 00 00 00 00 00 00 00 4e 00 00 00 00 00 00 00 |`.......N.......|
00000050 ae 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 4e 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |N...............|
00000070 45 16 01 00 00 00 00 00 76 da 44 2b 43 5f b5 bd |E.......v.D+C_..|
00000080 08 7b d2 b0 2e 16 7d 86 46 75 7b 79 f0 36 75 3b |.{....}.Fu{y.6u;|
00000090 a1 14 22 f3 0c 77 85 3c 70 00 61 00 72 00 2d 00 |.."..w.<p.a.r.-.|

hexdump -C Testing.docx.par2 | head
00000000 50 41 52 32 00 50 4b 54 84 00 00 00 00 00 00 00 |PAR2.PKT........|
00000010 76 1f e0 a4 5a 32 e0 84 d9 e9 32 32 06 9f 03 ff |v...Z2....22....|
00000020 71 48 73 d5 59 c6 ae 7c c7 21 3d ba 8d e5 ea 04 |qHs.Y..|.!=.....|
00000030 50 41 52 20 32 2e 30 00 46 69 6c 65 44 65 73 63 |PAR 2.0.FileDesc|
00000040 5d 74 b5 3d 64 ae 1f d8 ae 41 f1 8c 2f 7a cc c1 |]t.=d....A../z..|
00000050 27 9b bc 61 46 21 4d 37 a3 c7 f2 07 b4 b8 df 81 |'..aF!M7........|

Pretty straightforward. The only thing that would have made it easier is if the first version used “PAR1”, but be glad they didn’t as that signature is used by another!

hexdump -C null_list.parquet | head
00000000 50 41 52 31 15 00 15 18 15 18 2c 15 02 15 00 15 |PAR1......,.....|
00000010 06 15 06 00 00 02 00 00 00 02 00 02 00 00 00 02 |................|
00000020 01 26 42 1c 15 02 19 25 00 06 19 38 09 65 6d 70 |.&B....%...8.emp|
00000030 74 79 6c 69 73 74 04 6c 69 73 74 04 69 74 65 6d |tylist.list.item|
00000040 15 00 16 02 16 3a 16 3a 26 08 3c 36 02 00 00 00 |.....:.:&.<6....|
00000050 15 02 19 4c 48 0c 61 72 72 6f 77 5f 73 63 68 65 |...LH.arrow_sche|
00000060 6d 61 15 02 00 35 02 18 09 65 6d 70 74 79 6c 69 |ma...5...emptyli|
00000070 73 74 15 02 15 06 4c 3c 00 00 00 35 04 18 04 6c |st....L<...5...l|
00000080 69 73 74 15 02 00 15 02 25 02 18 04 69 74 65 6d |ist.....%...item|
00000090 6c bc 00 00 00 16 02 19 1c 19 1c 26 42 1c 15 02 |l..........&B...|

Apache Parquet is a more modern format used to store column-oriented data. At least they used a unique file extension!

Another common bit of software which uses the PAR extension is Solid Edge by Siemens. They use the PAR extension to encode their 3D parts format. For some reason this format still uses the OLE compound object container.

7z l tinyscrew.par 

Path = tinyscrew.par
Type = Compound
Physical Size = 86528
Extension = compound
Cluster Size = 512
Sector Size = 64

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
..... 31964 32256 PSMcluster0
..... 12 64 Versions
2001-12-19 15:44:14 D.... Display
2001-12-19 15:44:14 D.... ACIS
..... 8462 8704 ACIS/Solid1.sab
..... 238 256 PSMroots
2001-12-19 15:44:14 D.... Display/Cache0
2001-12-19 15:44:14 D.... Display/Styles
..... 1725 1728 Display/Styles/Library0
..... 12 64 Display/Styles/DefaultStyles
..... 88 128 Display/Cache0/Info
..... 4248 4608 Display/Cache0/L1-T1
..... 8 64 JSitesList
2001-12-19 15:44:14 D.... PARASOLID
..... 3389 3392 PARASOLID/STREAM434.D_B
..... 10402 10752 PARASOLID/STREAM434.P_B
..... 4 64 DocVersion2
..... 199 256 PSMclustertable
..... 8 64 PSMuserroots
..... 512 512 JVisibleData
2001-12-19 15:44:14 D.... PSMspacemap
..... 66 128 PSMspacemap/0x00002000
..... 6090 6144 PSMspacemap/0x00000000
..... 174 192 PSMspacemap/0x00004000
..... 4716 5120 PSMtypetable
..... 8 64 FamilyMembers
..... 8 64 BuildVersions
..... 150 192 PartsLiteData
..... 596 640 [5]C3teagxwOttdbfkuIaamtae3Ie
..... 476 512 [5]SummaryInformation
..... 12 64 PSMsegmenttable
..... 96 128 MSConvertedPropertyset
..... 148 192 [5]K4teagxwOttdbfkuIaamtae3Ie
..... 280 320 [5]DocumentSummaryInformation
..... 116 128 [5]SszbwomgY1udb2whAaq5u2jwCg
..... 264 320 [5]Rfunnyd1AvtdbfkuIaamtae3Ie
..... 140 192 Dynamic Attributes Metadata
..... 458 512 Unclustered Dynamic Attributes
------------------- ----- ------------ ------------ ------------------------
2001-12-19 15:44:14 75069 77824 32 files, 6 folders

We will have to use the a container signature to correctly identify this format. There are also ASM and DFT formats which are also Solid Edge formats which use the same OLE container. Hopefully there are some unique features we can use to identify them.

One other file format which uses the PAR extension is not listed in any of the registries. Not in PRONOM, TrID, Wikidata, or others. I came across it while researching another format, DVD Studio Pro. On a Macintosh computer running the now discontinued DVD Studio Pro, one could save their DVD mastering project as a “file” which used the DSPPROJ extension. I use the term file loosely here as it wasn’t actually a file, it was a folder with an extension which MacOS would interpret as a single file. These are the package formats Apple used and still uses quite frequently. Moving this folder to another other system results in a folder of content.

tree sample.dspproj 
/sample.dspproj
└── Contents
├── PkgInfo
└── Resources
├── Audio
├── MPEG
├── Menu
├── ModuleDataB
├── ObjectDataB
├── Openers.plist
├── Overlay
├── Picture
├── Render Data
│   ├── C4272B0100797459.M2V
│   └── PAR
│   └── C4272B0100797459.M2V.par
├── Styles
├── Temp
├── Templates
└── Thumbnails

14 directories, 6 files

This PAR extension is explained in the DVD Studio Pro manual:

About the Parse Files
To use an asset in a project, DVD Studio Pro needs to know some general information about it, such as its length, type, and integrity. Video assets encoded within DVD Studio Pro can include this information in the encoded files, or can create separate files for it. Assets encoded by Compressor outside of DVD Studio Pro can include this information if you select the “Add DVD Studio Pro meta-data” option in the Extras pane of the Encoder settings.
Assets encoded with other encoders, or with the “Add DVD Studio Pro meta-data” option disabled when using Compressor, must be parsed before DVD Studio Pro can use them. Parsing creates a small file, with the same name as the video asset and a “.par” extension that contains the required information. The parse file can take from several seconds to several minutes to create, depending on the size of the asset file.

hexdump -C E4712E541A60E300.M2V.par | head
00000000 56 50 41 52 00 00 00 20 00 00 00 00 00 01 e2 40 |VPAR... .......@|
00000010 00 00 00 00 00 c6 19 7c 2f 55 73 65 72 73 2f 74 |.......|/Users/t|
00000020 79 6c 65 72 2f 44 6f 63 75 6d 65 6e 74 73 2f 46 |yler/Documents/F|
00000030 69 6e 61 6c 20 52 65 6e 64 65 72 20 66 6f 72 20 |inal Render for |
00000040 44 56 44 20 56 51 42 2f 56 61 72 73 69 74 79 51 |DVD VQB/VarsityQ|
00000050 42 20 44 56 44 2f 56 61 72 73 69 74 79 51 42 2d |B DVD/VarsityQB-|
00000060 44 69 73 63 32 2e 64 73 70 70 72 6f 6a 2f 43 6f |Disc2.dspproj/Co|
00000070 6e 74 65 6e 74 73 2f 52 65 73 6f 75 72 63 65 73 |ntents/Resources|
00000080 2f 52 65 6e 64 65 72 20 44 61 74 61 2f 45 34 37 |/Render Data/E47|
00000090 31 32 45 35 34 31 41 36 30 45 33 30 30 2e 4d 32 |12E541A60E300.M2|

Parity, Parts, and Parse files, oh my.

If you thought we were done, you would be wrong! Let’s look at yet another PAR format.

hexdump -C MESSROH.PAR | head
00000000 08 69 64 73 32 30 30 30 30 d0 4e 01 51 46 42 00 |.ids20000.N.QFB.|
00000010 98 d0 4e 01 80 01 58 01 b6 b9 f7 bf 82 30 00 00 |..N...X......0..|
00000020 dc 08 00 00 60 51 f2 bf 82 30 01 59 ff ff ff ff |....`Q...0.Y....|
00000030 a4 d0 4e 01 28 3e f2 bf 78 63 a4 01 dc 08 00 0b |..N.(>..xc......|
00000040 5a 45 52 4f 2d 4f 46 46 53 45 54 01 18 0e ac 01 |ZERO-OFFSET.....|
00000050 d4 d0 4e 01 00 ac 43 00 18 0e ac 01 d4 d0 4e 01 |..N...C.......N.|
00000060 51 46 42 00 ec d0 4e 01 d4 00 4e 01 b6 b9 f7 bf |QFB...N...N.....|
00000070 5c 4c 75 81 5c 81 00 00 45 07 41 00 c0 0a 00 01 |\Lu.\...E.A.....|
00000080 cd d0 41 00 d5 d0 41 00 5c 81 00 00 dc 0a a4 01 |..A...A.\.......|
00000090 5b 5d 42 00 cc d0 4e 01 72 5d 42 00 7a 5d 42 00 |[]B...N.r]B.z]B.|

hexdump -C DUMMYDAT.PAR | head
00000000 08 73 65 69 73 6d 69 63 31 00 00 00 00 00 00 00 |.seismic1.......|
00000010 00 00 00 00 00 01 58 00 00 00 00 00 00 00 00 00 |......X.........|
00000020 00 00 00 00 00 00 00 00 00 00 01 59 00 00 00 00 |...........Y....|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0a |................|
00000040 41 4b 55 53 54 49 4b 4c 4f 47 00 00 00 00 00 00 |AKUSTIKLOG......|
00000050 00 00 00 00 02 2f 2f 00 08 41 47 43 2d 47 41 49 |.....//..AGC-GAI|
00000060 4e 00 00 00 00 00 00 00 00 00 00 00 00 32 00 00 |N............2..|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

This PAR format is called “Reflexw data-format“. This is a RAW format header that always is paired with a DAT file, together used to store geophysical wave data from devices such as GPR. Relexw is software made by Sandmeier geophysical research.

The PAR file samples I have don’t seem to have a consistent header as each have a unique set of bytes, but all of them have some similar bytes later in the file at around the 0x1D8 (472) offset:

000001d0  00 00 a0 3d 00 00 a0 41  00 00 00 00 00 00 00 00  |...=...A........|
000001e0 0a d7 23 3c 00 00 80 3f 00 00 00 00 00 00 00 00 |..#<...?........|
000001f0 00 00 00 00 cc cc dc 40 00 00 00 00 00 00 00 00 |.......@........|
00000200 00 00 80 3f 00 00 00 00 00 00 00 00 00 00 00 00 |...?............|
00000210 00 00 00 00 00 00 00 00 17 b7 d1 38 00 00 00 00 |...........8....|

It seems these sequence of bytes are the only consistent bytes among all my samples. I have no idea what they mean or reference. The specification does indicate some bytes which should lead to proper identification, but the integer used for the “HeaderMarker” is looking for a 4 byte “00 00 00 01”, which won’t be enough to cleanly identify the format. Love to hear what others can see from the spec. You can find some samples files here.

So we have some Parity files, Parts files, Parse files, Parquet files, and a Header file. I am sure other will be found and added to this lot. Hopefully the PAR files you run across will match one of these patterns! I am still working on a signature proposal. Stay Tuned!

HFE

Last week I had the pleasure of attending the 20th annual iPres conference on Digital Preservation in Ghent, Belgium. I enjoyed hearing from many of my respected colleagues on many aspects of preservation including one of my favorite topics, floppy disks. There was tutorials, lightning talks, and even a workshop, presented by Leontien Talboom, Elizabeth Kata, Chris Knowles, and myself. We titled the workshop “A Guide to Imaging Obscure Floppy Disk Formats“. The workshop was conceived by a mutual interest in imaging Wang 5.25in word processor disks, but expanded to include imaging of Amstrad 3in disks, 240K Brother Typewriter Disks, and Macintosh 400/800k disks.

I brought my hand soldered FluxEngine board and others brought their Greaseweazle board to show off how imaging obscure and uncommon disks can be done on a budget.

Photo of workshop taken on a Mavica Floppy Disk camera
Image taken during workshop on a Mavica FD200 Floppy Disk Camera.

During the conference we talked a bit about the different type of hardware that can be used and the difference between a disk image and flux image. There seems to be quite the exhaustive list of different types of file formats, some specific to a platform and others more generic. I recently did a blog post on the formats used by the Applesauce software, which have some unique features.

There are many disk image types which should be researched and added to PRONOM and other format description sites, but today lets take a look at a generic format used by many tools.

The HxC Floppy Emulator file format which the extension HFE is a popular format used with floppy drive emulators. There is a lot of complexity with what is included in many of these image formats, some are simply a raw sector representation of the binary data on a disk, others contain the complete flux readings from a floppy disk. The HFE format contains a little more than a raw image, including a header, a track lookup table, and the bitstreams for each track all with the purpose of emulating the physical media. The HFE format contains only a single pass over the data, where other formats may contain multiple reading of each track to get more complete data which can be helpful for damaged or purposely copy-protected disks. You can read more on Ashley’s blog, Library of Congress format description.

HFE version list

When using the HxC Floppy Emulator software, you can open and save to many different formats. The main format being their HFE native format. It comes in 5 versions.

hexdump -C test01.hfe | head
00000000 48 58 43 50 49 43 46 45 00 53 02 00 e8 01 00 00 |HXCPICFE.S......|
00000010 07 01 01 00 ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|

Above is a hexdump of the main SDCard HxC Floppy Emulator file format. The format specification shows the 8 byte header “HXCPICFE”. This is a very unique pattern and should be all we need to make a robust signature for the format, but we do need to take into account the other HFE “versions” and see if they might clash or need to be identified separately.

hexdump -C test02-a2.hfe | head 
00000000 48 58 43 50 49 43 46 45 00 53 02 00 d0 03 00 00 |HXCPICFE.S......|
00000010 07 01 01 00 ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|

The “A2” version of the format has the same header but some different bytes further into the file.

hexdump -C test03-rev2.hfe | head
00000000 48 58 43 50 49 43 46 45 01 53 02 00 00 00 00 00 |HXCPICFE.S......|
00000010 07 01 01 00 ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|

The “Rev 2” version also has the same header. But if you look at the 9th byte you can see the value changed from 00 to 01, which according to the specification, this is the revision byte.

hexdump -C test04-rev3.hfe | head 
00000000 48 58 43 48 46 45 56 33 00 53 02 00 e8 01 00 00 |HXCHFEV3.S......|
00000010 07 01 01 00 ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|

With “Rev 3” we see a change in the header with “HXCHFEV3” which appears to be referred to as HFEv3.

hexdump -C test05-stream.hfe | head 
00000000 48 78 43 5f 53 74 72 65 61 6d 5f 49 6d 61 67 65 |HxC_Stream_Image|
00000010 00 00 00 00 00 00 00 00 00 18 00 00 00 02 00 00 |................|
00000020 00 1a 00 00 53 00 00 00 02 00 00 00 40 9c 00 00 |....S.......@...|
00000030 07 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

This last format seems to be a special HxC stream image.

It seems the best option is to make three signatures to identify the three main headers. Additional software can be used to further parse the disk image. If you would like to see some sample images, you can download a bunch here. You can also take a look at my GitHub repository to see additional samples and a proposed set of signatures.

ATRAC

The year was 2001 and I found myself in need of an audio player and recorder. I had been burning CD’s for a few years, making mixed CD’s was fun and convenient, but I needed more flexibility. After some research I decided on a device that was super popular outside the United States, but was gaining some loyal fans.

This MZ-G750 MiniDisc device could record in a standard high quality mode through RCA, optical digital cable, and an optional microphone in mini-plug. This model also had the LP2 and LP4 modes which compressed higher, but could record up to 320 minutes on one MD disc.

Sony accomplished this by using a propriety compression codec called ATRAC, or Adaptive TRansform Acoustic Coding. This compression format was used with the MiniDisc and other Sony devices like the the flash memory Walkman’s sold later.

I recorded and stored a lot of music on the few disc’s I purchased over the next year, but as you may have surmised, the iPod came out later that year. I waited a bit but eventually purchased the updated 10GB model and the MiniDisc only was used to make a few recordings over the next little while.

As good as the MiniDisc is, the model I owned could record in a digital format, but lacked the connections to transfer the audio to a computer unless you used the optical cable and captured in real time to a computer with an optical input. This was by design, even when they put USB ports on later models, the software only allowed sending audio to the MiniDisc, but not back from the device.

A few years back I heard of some work the community has done to bring MiniDisc’s back from shadows. Now there is a thriving market and some models can cost a pretty penny. With that came some great tools and the ability to copy from the device back to the computer. The only problem, my device lacks a USB port. I kept my eye out for a “good” deal on a NetMD MiniDisc device. It took some time, but I am happy to report I am now the proud owner of a MZ-N420D.

With a new USB capable NetMD in hand, lets take a look at the different ATRAC formats!

The most common ATRAC formats are the ATRAC3 versions which generally have the extension OMA or OMG. But let’s start with ATRAC1, the format used on my earlier MiniDisc device when captured in Standard Mode. Using the amazing https://webmd.pro/ tool, I was able to connect my new device and “archive” my disc.

hexdump -C Test1.aea | head
00000000 00 08 00 00 54 65 73 74 31 00 00 00 00 00 00 00 |....Test1.......|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000100 00 00 00 00 1e 01 00 00 02 00 00 00 00 00 00 00 |................|
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000b50 0c a0 45 57 54 44 32 35 41 44 22 34 32 24 13 23 |..EWTD25AD"42$.#|
00000b60 32 23 22 12 11 11 11 11 76 18 69 75 f8 63 69 a7 |2#".....v.iu.ci.|
00000b70 a4 5d 46 22 45 36 1f 59 55 9d 41 55 19 51 45 17 |.]F"E6.YU.AU.QE.|
00000b80 45 14 55 38 c2 cb 2c b2 88 26 fd b2 17 b3 f0 0f |E.U8..,..&......|

ffprobe -i Test1.aea
[aea @ 0x7fc5e6c04fc0] Estimating duration from bitrate, this may be inaccurate
Input #0, aea, from 'Test1.aea':
Duration: 00:00:01.63, bitrate: 302 kb/s
Stream #0:0: Audio: atrac1, 44100 Hz, stereo, fltp, 292 kb/s

ATRAC1 files can have the AEA extension, which ffmpeg can decode, but MediaInfo doesn’t appear to have added the support. According to the decoder the magic numbers for the ATRAC1 format are “Magic is ‘00 08 00 00‘ in little-endian”. This pattern matches my files, but the recent addition PRONOM fmt/1968 doesn’t match all the samples I have.

The magic numbers are too simple to be the only pattern used in a signature. The Track title follows the magic numbers but are not static. Then there are quite a bit of zero bytes, like a lot. All the samples I have seem to have some data around the 260 offset, then more zero bytes until around 2400 to 2800 byte offset range. I scanned all the samples I have through Tridscan, and it looks like the only bytes in common are the magic header, lots of zero’s, and a few strings.

	<GlobalStrings>
<String>ED33</String>
<String>EUD3</String>
<String>FTDC</String>
<String>T322</String>
<String>TC32</String>
<String>TC43</String>
<String>UC22</String>
<String>UED3</String>
<String>VD33</String>
<String>VETC</String>
<String>WEDD</String>
</GlobalStrings>

The ffmpeg libavformat code does tell us at byte 264 there will be a 01 or 02 which indicates channels. 44.1 kHz is assumed and the bitrate is calculated from a constant by how many channels, so not much else to identify common patterns. More testing needed.

ATRAC3 is what allowed my original MiniDisc to record in LP2 and LP4, extending the recording time. This format was also how some DRM was added to the device and computer to allow for some checking-in and checking-out of files, but to control their use. This was done with Desktop software from Sony, originally in the form of the title SonicStage, later incorporating OpenMG to manage the DRM. I used SonicStage to encode some audio into OMG and OMA formats.

OpenMG format files

These are audio files which have been converted to ATRAC3 format and encrypted in OpenMG format, which is the copyright protection technology for audio contents specific to OpenMG (with the extension .omg).

hexdump -C 01-Untitled.omg | head
00000000 30 80 30 80 06 07 66 6f 70 65 6e 4d 47 02 02 03 |0.0...fopenMG...|
00000010 eb 04 14 01 0f 50 00 00 04 00 00 00 ba d0 90 49 |.....P.........I|
00000020 3d 7f 61 7b 91 c4 30 06 02 67 01 02 02 3f 00 06 |=.a{..0..g...?..|
00000030 02 68 01 02 04 00 59 47 80 02 01 00 02 03 02 03 |.h....YG........|
00000040 a0 02 02 01 80 02 01 00 00 00 04 08 f5 94 79 c9 |..............y.|
00000050 6b 78 75 22 04 84 00 59 5e 30 83 0b 71 39 e3 e8 |kxu"...Y^0..q9..|
00000060 27 29 00 00 00 00 00 00 00 00 26 e2 65 d0 de e0 |')........&.e...|
00000070 69 19 73 45 1c c4 3b 36 8d 02 3b 72 bd eb 84 df |i.sE..;6..;r....|
00000080 cd 20 4e 43 d3 e3 23 8a 3f 9e df 80 f1 86 d1 aa |. NC..#.?.......|
00000090 2b 93 bf 09 59 0d d6 8f 78 5d 45 3a 9f d8 79 8b |+...Y...x]E:..y.|

ffprobe -i /01-Untitled.omg
[oma @ 0x7fed2440e980] Format oma detected only with low score of 1, misdetection possible!
[oma @ 0x7fed2440e980] Couldn't find the EA3 header !
/01-Untitled.omg: Invalid data found when processing input

The good news is there appears to be a standard header for the OMG format, but ffmpeg assumes they are OMA files. Turns out OMG was the original form of the format, but was replaced with OMA starting with SonicStage v2.1.

hexdump -C 01-Untitled.oma | head
00000000 65 61 33 03 00 00 00 00 17 76 54 49 54 32 00 00 |ea3......vTIT2..|
00000010 00 17 00 00 02 00 55 00 6e 00 74 00 69 00 74 00 |......U.n.t.i.t.|
00000020 6c 00 65 00 64 00 28 00 31 00 29 54 41 4c 42 00 |l.e.d.(.1.)TALB.|
00000030 00 00 11 00 00 02 00 55 00 6e 00 74 00 69 00 74 |.......U.n.t.i.t|
00000040 00 6c 00 65 00 64 54 58 58 58 00 00 00 17 00 00 |.l.e.dTXXX......|
00000050 02 00 4f 00 4d 00 47 00 5f 00 54 00 52 00 41 00 |..O.M.G._.T.R.A.|
00000060 43 00 4b 00 00 00 31 54 58 58 58 00 00 00 25 00 |C.K...1TXXX...%.|
00000070 00 02 00 4f 00 4d 00 47 00 5f 00 41 00 4c 00 42 |...O.M.G._.A.L.B|
00000080 00 4d 00 53 00 00 00 55 00 6e 00 74 00 69 00 74 |.M.S...U.n.t.i.t|
00000090 00 6c 00 65 00 64 54 58 58 58 00 00 00 23 00 00 |.l.e.dTXXX...#..|
*
00000c00  45 41 33 03 00 60 ff 80  00 00 00 00 01 0f 50 00  |EA3..`........P.|
00000c10  00 04 00 00 00 60 8a 07  e3 0a c9 91 63 46 c6 bc  |.....`......cF..|
00000c20  22 52 03 76 00 05 66 48  00 00 3b 86 00 00 00 00  |"R.v..fH..;.....|
00000c30  00 00 20 30 00 00 00 00  00 00 00 00 00 00 00 00  |.. 0............|
00000c40  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

ffprobe -i 01-Untitled.oma
Input #0, oma, from '01-Untitled.oma':
Metadata:
title : Untitled(1)
album : Untitled
OMG_TRACK : 1
OMG_ALBMS : Untitled
OMG_ASGTM : 2366000
OMG_TIT2S : Untitled(1)
TLEN : 353000
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Audio: atrac3al ([34][0][0][0] / 0x0022), 44100 Hz, stereo, fltp

We learned from trying an OMG file in ffprobe that ffmpeg is looking for EA3 header, which is found in this OMA file. Both of these formats should have a nice header to work from for a signature. In fact there has already been a request and signature submitted for the OMA format. Mine are slightly different, but only takes a small tweak to work with all my samples. Also, it seems the extension AA3 was used for awhile before settling on OMA. OMA can have a few different types:

ffprobe -i 02-Untitled.oma 
[oma @ 0x7fbc7ef047c0] Estimating duration from bitrate, this may be inaccurate
Input #0, oma, from '/Star Trek/02-Untitled.oma':
Metadata:
title : Untitled(2)
album : Star Trek
OMG_TRACK : 2
OMG_ALBMS : Star Trek
OMG_ASGTM : 2366000
OMG_TIT2S : Untitled(2)
TLEN : 27000
Duration: 00:00:27.21, start: 0.000000, bitrate: 193 kb/s
Stream #0:0: Audio: atrac3p ([1][0][0][0] / 0x0001), 44100 Hz, stereo, fltp, 192 kb/s

I’ll leave the technical properties to be handled by tools more suited for parsing the format like ffmpeg. Maybe MediaInfo could have the formats added, but until then, it might be best to simply identify the main format. I am also aware of some later additions to the ATRAC family, such as ATRAC3plus, ATRAC Advanced Lossless, and ATRAC9 (WAV RIFF). There are other extensions like AT3 out there which use the ATRAC codec, like Sony’s Playstation or PSP. I will have to keep my eyes out for the even more elusive Hi-MD MiniDisc devices to find out more. For now, take a look at some samples and my proposal for signatures on my GitHub.

Worldox

Most File Systems have unique ways for doing things, but also many things in common. On a Macintosh you might have some extended attributes, or that pesky hidden .DS_Store file no one really knows why it’s there. On Windows you may find a hidden thumbs.db file throwing off your file count. Hidden files are everywhere. Many have a real purpose, and that purpose may be insignificant or important in finding or giving context to other files.

While processing a collection from a USB drive the other day, I came across a few files I hadn’t seen before. They were hidden files nestled in with a few folders of PDF’s. They have a unique name, so I figured it would be easy to find some documentation on them on the web. Turns out, there is very little.

-rwx------@ 1 tyler  staff    235 Aug 22 00:04 XNAME.CRS
-rwx------@ 1 tyler staff 235 Aug 22 00:04 XNAME.LIB

The files were only a couple years old, so I figured there had to be some modern software which created them. A look inside the files with a hex editor didn’t provide much information.

hexdump -C XNAME.LIB 
00000000 22 80 21 36 00 00 00 00 00 00 00 00 00 00 00 00 |".!6............|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000090 00 00 00 00 00 00 4c 3c 55 6e 61 73 73 69 67 6e |......L<Unassign|
000000a0 65 64 3e 00 00 00 00 00 00 00 00 00 00 00 00 00 |ed>.............|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

I was about to give up since there wasn’t much data and since they were hidden files, I assumed they were probably just some cached files with little value. But wanting to learn more I did some more digging and at first thought they might have something to do with DropBox, as a user said they just showed up one day, but later found they probably were created by some Document Management software known as Worldox. I found a support page claiming these two files are part of a database.

XNAME.LIB	Contains document numbers (DOS names), extended names, and file security information.
XNAME.CRS Contains custom profile field and version control information.

There is a key term in the definition of XNAME.LIB, “extended names”. I was curious what that meant and found Worldox has been around awhile. The World Software Corporation has been around since 1988 and Worldox was released in 1993, but before that it specialized in an interesting DOS software package called “Extend-A-Name” or “Extend-A-File”. The name gives away its purpose, it literally extends the name of the limited 8 Characters you could use in DOS. I can remember trying to decide on a filename that would accurately describe my file so I knew what it was later on. 8 characters is not enough to explain the content of a file, especially if you have hundreds or thousands of file to manage.

Extend-a-file was software which bonded with another piece of software like WordPerfect and loaded itself in memory. Then when you went to create a file or locate a file within WordPerfect, Extend-a-File would take over and allow you to create a file with a traditional 8 Character name, but also a name much longer.

This extended name allowed you to describe the files content with much more detail. Making it also very easy to find previous documents.

Pretty slick, this software really would make a big difference to managing a large amount of files in the old DOS days. Ok, it adds extended names, but where is this information stored? That is where the XNAME files come into play.

hexdump -C XNAME.LIB | head
00000000 6d 92 15 59 47 47 15 00 00 00 00 00 00 00 00 00 |m..YGG..........|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000090 00 00 00 00 00 00 4c 10 20 4d 41 53 54 45 52 20 |......L. MASTER |
000000a0 4c 49 42 52 41 52 59 20 2d 20 41 6c 6c 20 46 69 |LIBRARY - All Fi|
000000b0 6c 65 73 20 4c 69 73 74 65 64 00 20 64 72 69 76 |les Listed. driv|
000000c0 65 20 43 20 00 58 4e 50 4c 55 53 2e 24 24 24 00 |e C .XNPLUS.$$$.|
000000d0 fd 05 fe 49 6e 73 75 66 66 69 63 69 65 6e 74 20 |...Insufficient |
000000e0 64 69 73 6b 20 73 70 61 63 65 20 46 54 68 69 73 |disk space FThis|
000000f0 20 69 73 20 61 20 74 65 73 74 20 6f 66 20 58 4e | is a test of XN|
00000100 41 4d 45 20 20 20 20 20 20 20 20 20 20 20 20 20 |AME |
00000110 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 | |
00000120 20 20 20 20 20 20 20 20 00 56 44 53 30 30 30 30 | .VDS0000|
00000130 2e 44 4f 43 00 00 96 00 56 44 53 00 00 00 00 4f |.DOC....VDS....O|
00000140 55 30 30 30 30 30 30 0a 09 1d 00 02 00 00 00 00 |U000000.........|
00000150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

This XNAME.LIB was generated by a running copy of XNPLUS circa 1990, bonded with a copy of DisplayWrite 4. It adds much more information within the “Library”.

So it seems this method of storing the extended filenames and other metadata started in the Extend-a-File software and has been brought along and used in modern versions of the Worldox Document Management software. Much of its purpose to extend an 8 character filename to around 60 character is no longer needed as most systems now allow for filenames with at least 256 characters. I imagine there is more the software can add to these files, but found that the samples I have really don’t have any information in them at all. The Worldox software seems to be marketed toward law firms and others who have a lot of documents to manage, but I have been unable to find a way to play with the software to see what can be embedded within the XNAME.LIB files.

There is also some discussion out there about wether to backup these two hidden files and what might happen if they are lost. Regardless, you may want to think twice before tossing them as I almost did. They could contain valuable information needed to give context.

I am not sure it is possible to have a good signature for identification of these files. The samples I have and others I found online, here, here, here, and here, just don’t have much data within them. In fact they are all exactly 235 bytes. The only consistent byte within them and the samples I generated from XNPLUS is “4C” at offset 150, but everything else seems arbitrary. Here is a sample I generated from XNPLUS if you want to take a closer look.

A2R / MOOF / WOZ

There seems to be a never ending growing list of disk image formats. Many have features which are specific to the media and format. If you have ever imaged an older Macintosh floppy you know they are special. If you add in copy-protection which many early Apple II floppies have, and you need special drives, hardware, and a special format to store the floppy data.

When imaging special media, especially with unique media, it is best practice to image the floppies at the magnetic flux level.

Floppy disks contain magnetic fluctuations which are measured and recorded using specialized equipment. A popular method is using a Kryoflux board, floppy drive, and software. The software communicates with a custom controller board connected to a floppy drive through USB. If you are interested in the different controller boards, a good list has been compiled here.

A Kryoflux, fluxengine, greaseweazle, all can image specialized disks like a Macintosh 800k floppy, but the best controller board for them is an Applesauce setup. They are specifically designed to for the task. With that task, comes a few specialty formats.

A file format which can store flux data is a bit different than a regular disk image format. The flux data contains all the low-level recordings which can then be interpreted into disk images much like the original floppy. In the case of an Applesauce flux image, it can contain all the small nuances of the original floppy, this includes recording any copy protection or other creative methods used by software vendors throughout the years. The format used for storing this flux data is the A2R format.

A2R is in its third iteration. Let’s take a look at the basics of the format.

hexdump -C Samplev3.a2r | head
00000000 41 32 52 33 ff 0a 0d 0a 49 4e 46 4f 25 00 00 00 |A2R3....INFO%...|
00000010 01 41 70 70 6c 65 73 61 75 63 65 20 76 31 2e 38 |.Applesauce v1.8|
00000020 38 2e 35 20 20 20 20 20 20 20 20 20 20 20 20 20 |8.5 |
00000030 20 02 01 01 00 52 57 43 50 e9 49 6e 01 01 24 f4 | ....RWCP.In..$.|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 43 01 00 |.............C..|
00000050 00 01 27 3a 25 00 91 d9 00 00 21 20 21 21 21 21 |..':%.....! !!!!|
00000060 1f 21 21 21 21 1f 24 5e 24 1f 21 21 20 21 24 5c |.!!!!.$^$.!! !$\|
00000070 24 20 21 21 21 1f 24 5c 25 21 21 1f 21 21 23 5b |$ !!!.$\%!!.!!#[|
00000080 25 20 21 21 21 1f 21 22 23 3f 41 3f 26 3e 43 3f |% !!!.!"#?A?&>C?|
00000090 43 5f 41 27 3d 61 41 27 3d 61 3f 28 3e 61 3f 26 |C_A'=aA'=a?(>a?&|

hexdump -C Samplev2.a2r | head
00000000 41 32 52 32 ff 0a 0d 0a 49 4e 46 4f 24 00 00 00 |A2R2....INFO$...|
00000010 01 41 70 70 6c 65 73 61 75 63 65 20 76 31 2e 31 |.Applesauce v1.1|
00000020 2e 36 20 20 20 20 20 20 20 20 20 20 20 20 20 20 |.6 |
00000030 20 02 01 01 53 54 52 4d 75 17 5d 01 00 01 e6 da | ...STRMu.].....|
00000040 00 00 83 a9 12 00 12 1e 11 13 1e 13 1e 13 11 1f |................|
00000050 21 1f 11 13 1c 14 1e 30 14 20 1e 14 1e 14 1c 14 |!......0. ......|
00000060 1c 13 11 20 21 1f 11 11 0f 13 1e 14 1c 14 2e 21 |... !..........!|
00000070 13 1e 13 1e 14 1e 11 11 20 21 1f 11 11 13 1e 1f |........ !......|
00000080 13 20 30 21 11 11 0f 13 1e 13 11 30 1f 21 20 13 |. 0!.......0.! .|
00000090 11 30 1f 14 1e 30 14 1e 11 11 11 1e 13 11 1e 14 |.0...0..........|

The A2R format uses a chunk system to store the various pieces to the format. Earlier versions used a STRM Chunk to store all the raw flux data. Version 3 changed to a RWCP Chunk to store all the raw flux data. Applesauce uses a 2-pass imaging process, doing a rapid imaging to determine where on the media surface track data exists and then a second pass that captures longer durations for processing and error correction.

Once the full raw flux data has been captured that data can be interpreted as a disk image. The Applesauce software is able to make a regular disk image, a Disk Copy 4.2 file, which are well known and identify in PRONOM as fmt/625, but can also create a couple of special disk image formats which allow for special nuances on an original disk.

The WOZ Disk Image format is an offshoot of the Applesauce project. Capturing highly accurate bit data is of no use if you don’t have a container to hold the data. The WOZ format was designed to be able to contain every possible Apple ][ disk structure and layout. It can be so accurate that even copy protected software can’t tell that it isn’t an original disk.

The WOZ format has become very popular in the Apple II community and is ideal for emulating all the old games and software titles popular in the early 1980’s. You may have guessed where the name comes from. The internet archive has a large collection of WOZ disks in their WOZ-a-Day collection. The file format of a WOZ disk image is also a chunk based format similar to the A2R format, it has two versions. Let’s take a look.

hexdump -C WOZ 1.0/Blazing Paddles (Baudville).woz | head
00000000 57 4f 5a 31 ff 0a 0d 0a f6 f5 92 d6 49 4e 46 4f |WOZ1........INFO|
00000010 3c 00 00 00 01 01 00 01 01 41 70 70 6c 65 73 61 |<........Applesa|
00000020 75 63 65 20 76 30 2e 32 36 20 20 20 20 20 20 20 |uce v0.26 |
00000030 20 20 20 20 20 20 20 20 20 00 00 00 00 00 00 00 | .......|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 54 4d 41 50 a0 00 00 00 00 00 ff 01 01 01 ff 02 |TMAP............|
00000060 02 02 ff 03 03 03 ff 04 04 04 ff 05 05 05 ff 06 |................|
00000070 06 06 ff 07 07 07 ff 08 08 08 ff 09 09 09 ff 0a |................|
00000080 0a 0a ff 0b 0b 0b ff 0c 0c 0c ff 0d 0d 0d ff 0e |................|
00000090 0e 0e ff 0f 0f 0f ff 10 10 10 ff 11 11 11 ff 12 |................|

hexdump -C WOZ 2.0/Blazing Paddles (Baudville).woz | head
00000000 57 4f 5a 32 ff 0a 0d 0a 21 da c2 c8 49 4e 46 4f |WOZ2....!...INFO|
00000010 3c 00 00 00 02 01 00 01 01 41 70 70 6c 65 73 61 |<........Applesa|
00000020 75 63 65 20 76 31 2e 31 20 20 20 20 20 20 20 20 |uce v1.1 |
00000030 20 20 20 20 20 20 20 20 20 01 01 20 00 00 00 00 | .. ....|
00000040 0d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 54 4d 41 50 a0 00 00 00 00 00 ff 01 01 01 ff 02 |TMAP............|
00000060 02 02 ff 03 03 03 ff 04 04 04 ff 05 05 05 ff 06 |................|
00000070 06 06 ff 07 07 07 ff 08 08 08 ff 09 09 09 ff 0a |................|
00000080 0a 0a ff 0b 0b 0b ff 0c 0c 0c ff 0d 0d 0d ff 0e |................|
00000090 0e 0e ff 0f 0f 0f ff 10 10 10 ff 11 11 11 ff 12 |................|

Unlike a common disk image, a WOZ image contains more than the bits on the disk, it contains a mapping of all the tracks and the associated data, this is how it can even contain copy-protection usually only possible with a physical disk. The ‘TMAP’ chunk contains a track map and the ‘TRKS’ chunk contains all the data.

What the WOZ is for the Apple II, MOOF was made for the Macintosh. You may wonder what is with the funny name, but there is a long history around “Clarus the Dogcow”. I’m sure this factoid will help you impress your friends or win at trivia night. Again, the purpose of the special format for Macintosh disks is to allow for emulating disks, even with copy protection. You can also find quite the collection of old Macintosh software in the MOOF format on the Internet Archive, even emulate your favorite game, such as Dark Castle, which I played for hours as a kid. Also a chunk based format, let’s take a look at the header.

hexdump -C Dark Castle v1.0 - Disk 1.moof | head
00000000 4d 4f 4f 46 ff 0a 0d 0a b5 75 f9 4e 49 4e 46 4f |MOOF.....u.NINFO|
00000010 3c 00 00 00 01 01 00 01 10 41 70 70 6c 65 73 61 |<........Applesa|
00000020 75 63 65 20 76 31 2e 37 33 20 20 20 20 20 20 20 |uce v1.73 |
00000030 20 20 20 20 20 20 20 20 20 00 13 00 00 00 00 00 | .......|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 54 4d 41 50 a0 00 00 00 00 ff 01 ff 02 ff 03 ff |TMAP............|
00000060 04 ff 05 ff 06 ff 07 ff 08 ff 09 ff 0a ff 0b ff |................|
00000070 0c ff 0d ff 0e ff 0f ff 10 ff 11 ff 12 ff 13 ff |................|
00000080 14 ff 15 ff 16 ff 17 ff 18 ff 19 ff 1a ff 1b ff |................|
00000090 1c ff 1d ff 1e ff 1f ff 20 ff 21 ff 22 ff 23 ff |........ .!.".#.|

All three formats created for imaging and emulating Apple and Macintosh software are well documented and open. They are also well suited for preservation as they can contain extensive metadata in the INFO chunk which gives provenance information on the source of the files. The Applesauce software even has a camera to photograph the disk itself for archiving. All of this makes these formats great for preservation and emulation. Take a look at my proposal for a signature on my Github.

Binder

Microsoft is never in short supply of file formats. They have made many changes over the years. Introduced lots of products, some lasting longer than others. The list is quite long.

One such software was called Office Binder. Introduced with Office 95, it was a companion application to combine a number of OLE objects together in one “Binder”. Meant to be the digital version of an Office Binder one often uses for presentations or proposals.

You could add sections and include Word documents, Images, Powerpoint, Excel spreadsheets, basically any OLE object. Of course a Binder file itself was an OLE compound object. They had the extension OBD, and templates used OBT. The PRONOM registry has PUID’s for the different Binder versions, but there are some issues.

PUIDFormat NameFormat VersionExtension
fmt/237Microsoft Office Binder File for Windows95obd
fmt/240Microsoft Office Binder File for Windows97-2000obd
fmt/238Microsoft Office Binder Template for Windows95obt
fmt/241Microsoft Office Binder Template for Windows97-2000obt
fmt/239Microsoft Office Binder Wizard for Windows95obz
fmt/242Microsoft Office Binder Wizard for Windows97-2000obz
filename : 'Binder95-s01.obd'
filesize : 5120
modified : 2024-08-08T21:24:34-06:00
errors :
matches :
- ns : 'pronom'
id : 'fmt/240'
format : 'Microsoft Office Binder File for Windows'
version : '97-2000'
mime :
class :
basis : 'extension match obd; container name Binder with name only'

Turns out only one of the PRONOM PUID’s has an actual signature, the others are placeholders. So when I run Siegfried on an Office Binder 95 file, it comes back as fmt/240 which points to an Office Binder 97-2000 file. It’s a simple signature, looking for an internal file named “Binder”, which is inherent of all the Binder file types.

    <ContainerSignature Id="5500" ContainerType="OLE2">
<Description>Microsoft Office Binder File for Windows 97-2000</Description>
<Files>
<File>
<Path>Binder</Path>
</File>
</Files>
</ContainerSignature>

Taking a look inside the Office 95 Binder file, we can see the “Binder” file.

Path = Binder95-s01.obd
Type = Compound
Physical Size = 5120
Extension = compound
Cluster Size = 512
Sector Size = 64

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
..... 316 320 [5]SummaryInformation
..... 144 192 Binder
..... 280 320 [5]DocumentSummaryInformation
------------------- ----- ------------ ------------ ------------------------
740 832 3 files

hexdump -C Binder95-s01/Binder
00000000 90 00 00 00 05 00 00 00 00 00 00 00 05 00 00 00 |................|
00000010 00 00 00 00 a1 6a 8a 8e cc 55 ef 11 ab 06 00 0c |.....j...U......|
00000020 29 b1 b4 d0 00 00 00 00 00 00 00 00 00 00 00 00 |)...............|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 40 86 61 a6 |............@.a.|
00000040 0b ea da 01 00 00 00 00 00 00 00 00 40 86 61 a6 |............@.a.|
00000050 0b ea da 01 09 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 2c 00 00 00 00 00 00 00 01 00 00 00 |....,...........|
00000070 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000080 2c 00 00 00 2c 00 00 00 13 03 00 00 44 02 00 00 |,...,.......D...|

The bytes within a “Binder” file has some patterns, but nothing decipherable.

Microsoft Office Binder was only included in three versions of Office. Office 95, 97, and 2000. Let’s look at the other two versions.

Path = Binder97-s04.obd
Type = Compound
Physical Size = 5632
Extension = compound
Cluster Size = 512
Sector Size = 64

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
..... 28 64 HdrFtr
..... 144 192 Binder
..... 260 320 [5]SummaryInformation
..... 404 448 [5]DocumentSummaryInformation
------------------- ----- ------------ ------------ ------------------------
836 1024 4 files

Path = Binder2K-S01.obd
Type = Compound
Physical Size = 5632
Extension = compound
Cluster Size = 512
Sector Size = 64

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
..... 28 64 HdrFtr
..... 144 192 Binder
..... 260 320 [5]SummaryInformation
..... 232 256 [5]DocumentSummaryInformation
------------------- ----- ------------ ------------ ------------------------
664 832 4 files

It looks like version 97 and 2000 have an extra file. The “HdrFtr” file seems to reference a Header and Footer, which according to documentation was a feature added in Office 97.

What’s new in Office Binder 97

Office Binder makes it possible for you to group all your documents, workbooks, and presentations for a project in one place. To get started with Office Binder 97, add a new or existing document to your binder. Use the new Office 97 features while you work in a binder……. Print headers and footers for a binder

We can use the “HdrFtr” file within the container to differentiate between the 95 version and 97-2000 formats. Perhaps, a closer look at the DocumentSummaryInformation file in the future, might help with a more precise identification later. There doesn’t seem to be anything to distinguish an OBD file from a OBT template file, so those PUID’s may not be needed. The other format related to the Binder software has the OBZ extension. It is called a Wizard template file in some documentation, but I have been unable to find any type of “Wizard” functionality in the Office Binder Apps to generate a file. The OBZ format seems to have something to do with macros in Visual Basic. Luckily there are a few examples available on Office install disc‘s.

Path = CLIENT.OBZ
Type = Compound
Physical Size = 364032
Extension = doc
Cluster Size = 512
Sector Size = 64

Date Time Attr Size Compressed Name
------------------- ----- ------------ ------------ ------------------------
1995-07-05 17:25:15 D.... 7
1995-07-05 17:25:14 D.... 5
1995-07-05 17:25:13 D.... 4
..... 106 128 4/[1]CompObj
..... 20 64 4/[1]Ole
..... 8880 9216 4/WordDocument
..... 32 64 4/[3]View000
..... 492 512 4/[5]SummaryInformation
..... 236 256 4/[5]DocumentSummaryInformation
1995-07-05 17:25:14 D.... 6
..... 17760 17920 6/Book
..... 20 64 6/[1]Ole
..... 0 0 6/[3]View000
..... 102 128 6/[1]CompObj
..... 3260 3264 6/[5]SummaryInformation
..... 192 192 6/[5]DocumentSummaryInformation
..... 106 128 5/[1]CompObj
..... 20 64 5/[1]Ole
..... 8055 8192 5/WordDocument
..... 32 64 5/[3]View000
..... 7280 7680 5/[5]SummaryInformation
..... 220 256 5/[5]DocumentSummaryInformation
1995-07-05 17:25:16 D.... 9
1995-07-05 17:25:15 D.... 8
..... 13857 14336 8/Book
..... 20 64 8/[1]Ole
..... 0 0 8/[3]View000
..... 102 128 8/[1]CompObj
..... 188 192 8/[5]SummaryInformation
..... 196 256 8/[5]DocumentSummaryInformation
..... 854 896 Binder
1995-07-05 17:25:19 D.... 10
..... 80382 80384 10/Book
..... 20 64 10/[1]Ole
..... 0 0 10/[3]View000
..... 102 128 10/[1]CompObj
..... 4044 4096 10/[5]SummaryInformation
1995-07-05 17:25:19 D.... 10/_VBA_PROJECT
..... 9425 9728 10/_VBA_PROJECT/812f9922c6
..... 12302 12800 10/_VBA_PROJECT/7b2f9922a4
..... 36937 37376 10/_VBA_PROJECT/dir
..... 6609 6656 10/_VBA_PROJECT/7e2f9922b5
..... 23014 23040 10/_VBA_PROJECT/872f9922e8
..... 7995 8192 10/_VBA_PROJECT/842f9922d9
..... 5338 5632 10/_VBA_PROJECT/902f992333
..... 36119 36352 10/_VBA_PROJECT/8d2f99231e
..... 18129 18432 10/_VBA_PROJECT/932f992342
..... 13055 13312 10/_VBA_PROJECT/b42fbcaa59

..... 208 256 10/[5]DocumentSummaryInformation
..... 4228 4608 [5]SummaryInformation
..... 956 960 [5]DocumentSummaryInformation
..... 106 128 9/[1]CompObj
..... 20 64 9/[1]Ole
..... 5914 6144 9/WordDocument
..... 0 0 9/[3]View000
..... 1520 1536 9/[5]SummaryInformation
..... 220 256 9/[5]DocumentSummaryInformation
..... 16141 16384 7/Book
..... 20 64 7/[1]Ole
..... 0 0 7/[3]View000
..... 102 128 7/[1]CompObj
..... 188 192 7/[5]SummaryInformation
..... 192 192 7/[5]DocumentSummaryInformation
------------------- ----- ------------ ------------ ------------------------
1995-07-05 17:25:19 345316 351168 55 files, 8 folders

Sure enough, the OBZ file has a Visual Basic macro (VBA_Project). Unfortunately, it appears to be nested in an additional folder within the container, with a variable number number which is likely to change from file to file. That fact will make identification in PRONOM much more difficult, as the signatures are not designed for variable names. Possibly something we can investigate later.

Microsoft Binder was only released in Office 95, 97, and 2000, but was supported in Office XP and 2003 through an UNBIND.EXE application which would simply separate all the different objects back out to the individual files.

The Microsoft Office Binder is not included in Office 2003. However, if a Binder file created in a previous version of Office contains information you want to access, you can use the Unbind tool to pull out the information and save it in the formats of the appropriate programs. In order to do this procedure, the Unbind tool must be installed.

    As always, you can look at some sample files and my proposal for updated signatures on my GitHub page.

    UFO

    Researching file formats isn’t for everyone. Others may find it boring or even odd. Trying to explain to others the nuances of a binary format versus a container format would bring many tears. Their reactions sometimes are similar to hearing someone explain their belief in aliens. Passionate, but a bit on the crazy side.

    So with aliens and containers on my mind, let’s take a look at a format with the extension UFO. It is not an unidentified flying object or a UAP, it may as well be an unidentified file object, but in this case it is a “Ulead File for Objects” format. It is the exclusive file format for use with the PhotoImpact software from Ulead Systems, a Taiwanese developer known for many popular software programs. First released in 1996 with version 3, the PhotoImpact software was marketed as “a fully object-based tool, which pioneered a number of important innovations“.

    The reason it was a considered a full object-based tool was the UFO format is based on the, at the time, popular OLE Compound File Storage object format developed by Microsoft. So by using some OLE tools we can take a closer look at some of these Unidentified File Objects……..

    oleid Sample.ufo 
    oleid 0.60.1 - http://decalage.info/oletools
    THIS IS WORK IN PROGRESS - Check updates regularly!
    Please report any issue at https://github.com/decalage2/oletools/issues

    Filename: Sample.ufo
    --------------------+--------------------+----------+--------------------------
    Indicator |Value |Risk |Description
    --------------------+--------------------+----------+--------------------------
    File format |Generic OLE file / |info |Unrecognized OLE file.
    |Compound File | |Root CLSID: - None
    |(unknown format) | |
    --------------------+--------------------+----------+--------------------------
    Container format |OLE |info |Container type
    --------------------+--------------------+----------+--------------------------
    Encrypted |False |none |The file is not encrypted
    --------------------+--------------------+----------+--------------------------
    VBA Macros |No |none |This file does not contain
    | | |VBA macros.
    --------------------+--------------------+----------+--------------------------
    XLM Macros |No |none |This file does not contain
    | | |Excel 4/XLM macros.
    --------------------+--------------------+----------+--------------------------
    External |0 |none |External relationships
    Relationships | | |such as remote templates,
    | | |remote OLE objects, etc
    --------------------+--------------------+----------+--------------------------

    Well, it is a OLE file, but is unrecognized/unidentified by the oletools software. It also appears to be missing the root entry and CLSID you commonly find in OLE files. Since this is an OLE container we can also just use 7zip to peek inside as well.

    Path = Sample.ufo
    Type = Compound
    Physical Size = 937984
    Extension = compound
    Cluster Size = 512
    Sector Size = 64

    Date Time Attr Size Compressed Name
    ------------------- ----- ------------ ------------ ------------------------
    1999-05-25 03:33:05 D.... OS-3
    1999-05-25 03:33:04 D.... OS-1
    1999-05-25 03:33:03 D.... OS-0
    ..... 31122 31232 OS-0/ObjectImage
    ..... 1316 1344 OS-0/ObjectData
    ..... 137996 138240 OS-0/PathStream
    ..... 19591 19968 OS-0/ObjectMask0
    1999-05-25 03:33:05 D.... OS-2
    ..... 43405 43520 OS-2/ObjectImage
    ..... 1316 1344 OS-2/ObjectData
    ..... 176204 176640 OS-2/PathStream
    ..... 25524 25600 OS-2/ObjectMask0
    ..... 41588 41984 OS-1/ObjectImage
    ..... 1316 1344 OS-1/ObjectData
    ..... 170132 170496 OS-1/PathStream
    ..... 25221 25600 OS-1/ObjectMask0
    ..... 34505 34816 LtfMainImage
    ..... 656 704 LtfHeader
    1999-05-25 03:33:06 D.... OS-4
    ..... 19249 19456 OS-4/ObjectImage
    ..... 1316 1344 OS-4/ObjectData
    ..... 4842 5120 LtfPreviewImage
    ..... 1160 1216 LtfObjectList
    ..... 31753 32256 OS-3/ObjectImage
    ..... 1316 1344 OS-3/ObjectData
    ..... 131892 132096 OS-3/PathStream
    ..... 19439 19456 OS-3/ObjectMask0
    ------------------- ----- ------------ ------------ ------------------------
    1999-05-25 03:33:06 920859 925120 22 files, 5 folders

    In this sample file, we have a bunch of directories and objects, but none of what we expect to see in an OLE file, such as a “SummaryInformation” or “DocumentSummaryInformation” like we would see in a Word DOC file. By not having the standard contents of the container, it makes these files very specific to PhotoImpact software.

    Path = PhotoImpactX3-s01.ufo
    Type = Compound
    Physical Size = 5120
    Extension = compound
    Cluster Size = 512
    Sector Size = 64

    Date Time Attr Size Compressed Name
    ------------------- ----- ------------ ------------ ------------------------
    ..... 20 64 HotspotStream
    ..... 656 704 LtfHeader
    ..... 20 64 SliceInfoStream
    ..... 412 448 LtfPreviewImage
    ..... 714 768 WebPropStream
    ..... 20 64 ManualHotspotScriptInfoStream
    ..... 20 64 ObjectHotspotScriptInfoStream
    ------------------- ----- ------------ ------------ ------------------------
    1862 2176 7 files

    Here is another UFO file from the last version of the software PhotoImpact X3 when it was owned by Corel, but phased out in 2009. This is the basic file structure with no objects added to the file. We can be fairly confident these are the base files in most every other UFO file. It doesn’t have any of the “OS” folders which contain the objects, so I think the LtfHeader file might be our best bet for a signature. Let’s take a look at the Hex values for a few of them.

    hexdump -C PhotoImpactX3-s01/LtfHeader| head
    00000000 90 02 00 00 4c 54 46 00 58 02 00 00 02 00 ba dc |....LTF.X.......|
    00000010 ee 02 00 00 26 02 00 00 80 fc 0a 00 80 fc 0a 00 |....&...........|
    00000020 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
    00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

    hexdump -C Sample/LtfHeader| head
    00000000 90 02 00 00 4c 54 46 00 90 01 00 00 02 00 f7 bf |....LTF.........|
    00000010 90 01 00 00 90 01 00 00 80 fc 0a 00 80 fc 0a 00 |................|
    00000020 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 |................|
    00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

    hexdump -C v3/ANIMALS/LtfHeader| head
    00000000 90 02 00 00 4c 54 46 00 64 00 00 00 02 00 6e 00 |....LTF.d.....n.|
    00000010 40 01 00 00 c8 00 00 00 80 fc 0a 00 80 fc 0a 00 |@...............|
    00000020 00 00 00 00 11 00 00 00 01 00 00 00 01 00 00 00 |................|
    00000030 01 00 00 00 60 00 00 00 3c 00 00 00 60 00 00 00 |....`...<...`...|

    Making a signature using the first 8 bytes of the LtfHeader file appears to have worked for all the 3,400+ sample files I have collected. Problem is it also worked for another extension found in the some of the later versions of PhotoImpact.

    When you have successfully finished your template, make sure to save it in the Ulead File For Photo Project format (*.UFP). This allows you to open and use your template in the Photo Projects dialog box. In the Template tab, click Open Project and browse for the created file.

    They appear to be a template version for the format so we should be fine just adding the extension to the same signature.

    Well, this Unidentified File Object is no longer unidentifiable. Was it sent by aliens? Possibly, but at least we know where these UFO’s came from, PhotoImpact. Take a look at the samples and proposed signature in my GitHub.

    Also be sure to join us at this years iPres conference and attend our workshop on container signatures in PRONOM!

    ePic

    Image compression has been around for awhile. It seems everyone took a crack at making better algorithms to improve quality and size. Some chose to invent new ways and others chose to use existing methods but with their own flare. Kodak tried this with their PhotoCD, but there was a couple other photo processing options that popped up in 90’s. One was Seattle FilmWorks and another was Konica PC PictureShow. Both of which used “proprietary” formats to deliver developed film on disk.

    Seattle FilmWorks later called PhotoWorks, used an image format with the extension SFW and was based on BMP and JPG, but with their own twist. The same goes for the format used by Konica’s PC PictureShow.

    Konica PC PictureShow Disk

    If you took your film in to be developed at one of Konica’s photo labs, you could could have those images put on a diskette or later a CD-R. The disks came with software to view your photos called PC PhotoShow. The images stored on disk where in another proprietary format with the extension KQP. The KQP format was actually licensed from another company called Pegasus Imaging Corporation, later known as Accusoft. They developed their own way to compress a JPEG file which they called an ePic. An SDK called PICTools was offered for many years, but seems not to be available anymore.

    ePIC (Proprietary)
    • Supports PIC format compression, replacing the JPEG Huffman encoder with the proprietary ELS entropy encoder for 15% more compression.
    • Can be losslessly converted back to JPEG format using Op_RORE.

    A search on the internet for Konica KQP shows quite a few people over the years wondering what to do with their old disks and converting the old format to JPG, only to find a lack of information and available tools to do so. One such person used python to edit the file and making the file renderable as a JPG. While the method worked well for their KQP files, it might not work for all of them. Let’s look closer and understand why.

    hexdump -C Sample.PIC | head
    00000000 42 4d 00 00 00 00 00 00 00 00 42 04 00 00 44 00 |BM........B...D.|
    00000010 00 00 34 08 00 00 24 fa ff ff 01 00 18 00 4a 50 |..4...$.......JP|
    00000020 45 47 00 00 00 00 00 00 00 00 00 00 00 00 fc 00 |EG..............|
    00000030 00 00 ec 00 00 00 2c 00 00 00 18 00 00 00 00 00 |......,.........|
    00000040 00 00 02 00 00 00 08 00 00 00 01 00 00 00 01 00 |................|
    00000050 00 00 60 00 00 00 00 00 60 00 00 60 00 00 00 00 |..`.....`..`....|
    00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

    At first glance the file appears to be a Bitmap (BMP), and it does have a Bitmap header claiming to have JPEG compression, but if we look a little further into the file.

    identify -verbose Sample.PIC   
    identify: length and filesize do not match `Sample.PIC' @ error/bmp.c/ReadBMPImage/950.
    identify: unrecognized compression `Sample.PIC' @ error/bmp.c/ReadBMPImage/1019.

    hexdump -C Sample.PIC
    00000000 42 4d 00 00 00 00 00 00 00 00 42 04 00 00 44 00 |BM........B...D.|
    00000010 00 00 34 08 00 00 24 fa ff ff 01 00 18 00 4a 50 |..4...$.......JP|
    00000020 45 47 00 00 00 00 00 00 00 00 00 00 00 00 fc 00 |EG..............|
    00000030 00 00 ec 00 00 00 2c 00 00 00 18 00 00 00 00 00 |......,.........|
    00000040 00 00 02 00 00 00 08 00 00 00 01 00 00 00 01 00 |................|
    00000050 00 00 60 00 00 00 00 00 60 00 00 60 00 00 00 00 |..`.....`..`....|
    00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
    *
    00000400 00 00 60 00 00 00 00 00 60 00 00 60 00 00 00 00 |..`.....`..`....|
    00000410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
    *
    00000440 00 00 ff d8 ff e0 00 10 4a 46 49 46 00 01 02 02 |........JFIF....|
    00000450 00 00 00 00 00 00 ff e1 00 0a 50 49 43 00 01 19 |..........PIC...|
    00000460 1e 01 ff c0 00 11 08 05 dc 08 34 03 01 11 00 02 |..........4.....|

    We find a JPG marker, in fact almost the whole jpg file is included, except the quantization tables for luminance and chrominance which are needed to properly display the image. This is the area the Pegasus company thought they could encode better to further compress the image. Their method was to use a new algorithm called ELS (Entropy Logarithmic-Scale). This new method was used by the PICTools software to make a Pegasus PIC file while Konica used it for their KQP format. They are identical. By choosing the luminance and chrominance values during compression, you could make a highly compressed image, but required specific software to render.

    Pegasus also made use of a special custom APP marker (PIC) within the JPEG structure of the PIC/KQP and also any JPG compressed using their software. This marker which takes up around 8 bytes holds the luminance and chrominance values. Take the above sample for instance, it is compressing the image with a Luminance of 25 and a Chrominance of 30, these are integer values and in hex they would be “19” and “1E” respectively.

    hexdump -C Sample.PIC      
    00000440 00 00 ff d8 ff e0 00 10 4a 46 49 46 00 01 02 02 |........JFIF....|
    00000450 00 00 00 00 00 00 ff e1 00 0a 50 49 43 00 01 19 |..........PIC...|
    00000460 1e 01 ff c0 00 11 08 05 dc 08 34 03 01 11 00 02 |..........4.....|
    00000470 11 01 03 11 01 ff c4 00 51 00 01 00 03 01 00 00 |........Q.......|

    So in theory one could strip out any part of the file before the JPG beginning of file magic bytes (FF D8 FF E0), locate the APP marker, use the values to generate the two quantization tables, insert them in the appropriate spot and save out a JPG file.

    This may be the case for the first few versions of the ePic format, but later versions got more complicated. It seems a “PIC2” version replaced the earlier versions and this format is a little more complicated.

    hexdump -C Sample.KQP | head
    00000000 50 49 43 32 01 08 00 00 00 64 00 01 00 b9 3e 00 |PIC2.....d....>.|
    00000010 00 05 08 00 00 00 4a 50 47 45 03 00 00 00 16 24 |......JPGE.....$|
    00000020 00 00 00 43 6f 6d 70 72 65 73 73 69 6f 6e 20 62 |...Compression b|
    00000030 79 20 50 65 67 61 73 75 73 20 49 6d 61 67 69 6e |y Pegasus Imagin|
    00000040 67 20 43 6f 72 70 2e 06 68 3e 00 00 ff d8 ff e0 |g Corp..h>......|
    00000050 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 |..JFIF..........|
    00000060 ff e1 00 16 50 49 43 00 03 00 00 01 00 00 00 00 |....PIC.........|
    00000070 00 00 00 00 00 00 00 00 ff db 00 84 00 0f 0a 0a |................|
    00000080 0a 0a 06 0f 0a 0a 0a 0f 0f 0f 0f 14 1e 14 14 14 |................|
    00000090 14 14 28 1e 1e 19 1e 2d 28 32 32 2d 28 2d 2d 32 |..(....-(22-(--2|

    Instead of the Bitmap (BMP) header, a proprietary PIC2 header is used, still containing a JPG in the JFIF format along with a the PIC APP marker, but encoded in a way that the simple method of adding a quantization table may not work. With the original format the JPG and the PIC/KQP were approximately the same size, this new version significantly reduces the size of the PIC/KQP in comparison with the JPG.

    The ELS compression technology used in the ePic format seems to be patented by Pegasus and Accusoft, but is not entirely hidden as the libavcodec library includes a ELS decoder. Might be a fun project to use the code to decode the PIC/KQP formats fully.

    In the meantime, a signature identifying the two versions should be added to PRONOM. Check out my proposal on my GitHub. If you need to convert your KQP or PIC files back to JPG here are a few links:

    Konica PC PictureShow Version 4 (PIC2)

    Accusoft PICTools Apollo Demo (Windows 7 Compatible)

    Konica PC PictureShow for Macintosh

    Interactive Quicktime

    One of my favorite legacy formats to explore is any type of multimedia CD-ROM. The 1990’s and early 2000’s were filled with all sorts of multimedia for CD, Web, and Television. It is also one of the most difficult formats to try and preserve for the future. Many CD-ROM’s are filled with executables and/or Macromedia Director media, later having flash content. The operating systems and security needs today make playback almost impossible. For this reason many have built emulation services to mimic the original operation system and software to allow the many historic multimedia CD-ROM’s to once again interact with the user in a way many current systems still struggle with.

    Many CD-ROM’s would come as Hybrid disc’s allowing them to be used on a Windows and Macintosh system, sometimes providing two different experiences. Then there were CD-Extra or Enhanced CD‘s as a separate session to an Audio CD which would contain bonus content playable only on a computer.

    For fun I took a look back at some of my older Audio CD titles. I came across a couple, one claiming to be a “CD-Extra” and another an “Enhanced CD“. The CD-Extra disc when queried with cd-info claimed to have 12 tracks, with the 12th being a data XA track.

    Disc mode is listed as: CD-ROM Mixed
    CD-ROM Track List (1 - 12)
    #: MSF LSN Type Green? Copy? Channels Premphasis?
    1: 00:02:00 000000 audio false no 2 no
    2: 02:13:66 009891 audio false no 2 no
    3: 05:21:28 023953 audio false no 2 no
    4: 08:18:19 037219 audio false no 2 no
    5: 12:28:37 055987 audio false no 2 no
    6: 16:11:58 072733 audio false no 2 no
    7: 19:21:56 086981 audio false no 2 no
    8: 23:17:49 104674 audio false no 2 no
    9: 26:01:17 116942 audio false no 2 no
    10: 28:30:02 128102 audio false no 2 no
    11: 31:07:70 139945 audio false no 2 no
    12: 37:29:46 168571 XA true no
    170: 51:35:07 231982 leadout (520 MB raw, 516 MB formatted)
    CD Analysis Report
    CD-Plus/Extra
    session #2 starts at track 12, LSN: 168571

    Mounting the 12th track showed a mix of Macromedia Director (.DIR) files and quite a few Quicktime MOV movies. Playback was not possible on my current computer so I had to resort to using an emulator to experience this bonus content, full of band member photos and biographies.

    The other disc I pulled out to explore was a bit different. Using cd-info the disc looked very similar:

    Disc mode is listed as: CD-ROM Mixed
    CD-ROM Track List (1 - 13)
    #: MSF LSN Type Green? Copy? Channels Premphasis?
    1: 00:02:00 000000 audio false no 2 no
    2: 04:20:08 019358 audio false no 2 no
    3: 08:04:27 036177 audio false no 2 no
    4: 11:15:62 050537 audio false no 2 no
    5: 14:54:32 066932 audio false no 2 no
    6: 19:57:73 089698 audio false no 2 no
    7: 26:12:36 117786 audio false no 2 no
    8: 29:51:59 134234 audio false no 2 no
    9: 34:44:00 156150 audio false no 2 no
    10: 39:36:62 178112 audio false no 2 no
    11: 42:06:01 189301 audio false no 2 no
    12: 45:42:26 205526 audio false no 2 no
    13: 57:10:54 257154 XA true no
    170: 72:56:67 328117 leadout (735 MB raw, 730 MB formatted)
    CD Analysis Report
    CD-Plus/Extra
    session #2 starts at track 13, LSN: 257154

    The disc’s, even though were labeled CD-Extra and Enhanced CD, had the same structure and format. The difference was in the type of multimedia used. There was a simple application which launched Quicktime and loaded a single MOV movie. But, this was not your regular Quicktime Movie, this is a highly complex Interactive Quicktime movie.

    The Quicktime movie could only be launched from an older operating system using Quicktime 6, and on the Macintosh, only a PPC CPU. The movie would launch with an interactive menu, allowing navigation as you might find on a DVD or Flash website, but all within a single MOV file. When I ran MediaInfo on the MOV file I got back quite a few tracks:

    <media ref="/Volumes/VOLCANOECD/ALECD.mov">
    <track type="General">
    <VideoCount>10</VideoCount>
    <AudioCount>1</AudioCount>
    <OtherCount>51</OtherCount>
    <FileExtension>mov</FileExtension>
    <Format>QuickTime</Format>
    <Format_Settings>Compressed header</Format_Settings>

    Ten video tracks and 51 other tracks. Exploring with Quicktime, I could see the entire list of embedded content:

    Quicktime movies, an Audio track, dozens of Flash, Photos, Animations, Sprites, with the possibility of more. These types of Quicktime files had requirements in order to run with Quicktime 6 being the last which could playback all the content correctly. Current versions of Quicktime give a warning on the lack of compatibility.

    This Interactive Quicktime movie proudly claims; “Made with LiveStage Pro“, which was an authoring environment for Quicktime made by Totally Hip Software Inc. Started in 1995, but seemed to disappear after 2004 with no new development and by 2014 the website went offline.

    If you would like to see a couple of Apple created simple examples see here.

    LiveStage Pro was a very powerful authoring tool in its time, another similar tool called Electrifier competed for the interactive Quicktime market. Adobe GoLive also competed, but offered fewer features. The final Quicktime movie exported from LiveStage Pro was the main component, but the software did save a project format with the extension “LSD”. Versions 2 through 4 of LiveStage Pro had a similar header.

    hexdump -C LiveStagePro4-s01.lsd | head
    00000000 4c 53 41 46 00 00 00 04 00 00 09 16 00 00 00 00 |LSAF............|
    00000010 00 00 00 00 00 00 00 00 00 00 09 0a 73 65 61 6e |............sean|
    00000020 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 18 |................|
    00000030 56 53 4e 6e 00 00 00 01 00 00 00 00 00 00 00 00 |VSNn............|
    00000040 00 00 00 04 00 00 08 84 4d 50 52 4e 00 00 00 01 |........MPRN....|
    00000050 00 00 00 49 00 00 00 00 00 00 00 21 6d 4f 55 54 |...I.......!mOUT|
    00000060 00 00 00 01 00 00 00 00 00 00 00 00 55 6e 74 69 |............Unti|
    00000070 74 6c 65 64 2e 6d 6f 76 00 00 00 00 18 57 6c 65 |tled.mov.....Wle|
    00000080 66 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |f...............|
    00000090 00 00 00 00 18 57 74 6f 70 00 00 00 01 00 00 00 |.....Wtop.......|

    All the samples from version 2 through 4 have the first four bytes as “LSAF“. It also seems the next four bytes may be version related. Version 1 however has a different header.

    hexdump -C contest.lsd | head
    00000000 4c 53 50 72 00 00 00 08 00 00 00 00 00 00 02 80 |LSPr............|
    00000010 01 e0 00 00 00 00 02 58 00 00 00 01 00 00 00 01 |.......X........|
    00000020 00 00 00 02 00 00 00 00 00 08 00 00 00 00 00 00 |................|
    00000030 00 00 08 53 02 d9 ff c9 04 76 02 97 01 00 44 00 |...S.....v....D.|
    00000040 0b 02 fb 03 c9 00 00 00 01 00 00 00 01 00 00 00 |................|
    00000050 00 07 41 63 74 69 6f 6e 73 00 00 00 00 00 00 00 |..Actions.......|
    00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
    00000070 00 00 00 00 00 00 00 00 05 00 00 00 01 50 49 43 |.............PIC|
    00000080 54 ff ff 00 00 c1 ff 03 72 65 64 65 6e 6e 41 79 |T.......redennAy|
    00000090 98 05 41 77 78 00 00 01 7a 00 10 00 00 31 fc 30 |..Awx...z....1.0|

    Identification of a LiveStage project should be simple enough, but identifying and rendering back a Quicktime movie made by this software takes some work. In fact there are many “Enhanced CD’s” and CD-Extra titles out there with quite a few system requirements. If we are not careful, many of these little gems might get more difficult to experience or lost completely.

    If you would like to explore the Quicktime Movie from the Enhanced CD mentioned here, send me a message. You can also take a look at my signature proposal and samples files on my Github for LiveStage.