RIS Citation

Posted on by Thor Leave a comment

Up until recently I was working in a Corporate archive preserving all sort of content. The corporation throughout the years used many different software packages to produce all sorts of data. When I moved to an academic library I saw much of the same content, but there was a some new file formats which I needed to document and manage. Many of those come from scholarly journals , theses, dissertation, and data sets for projects.

One format which I came across often but seems to be missing from the standard file format known lists was the RefMan citation format. This format is a simple text based format which serves to standardize citations from scholarly sources. Created by Research Information Systems, the format uses the RIS extension used by Procite and Reference Manager (RefMan). ISI ResearchSoft managed the format for a bit in the 1990’s, this is where you can find most of the specifications.

Now that I am a little more familiar with the format I see it everywhere! Find any scholarly journal and there will usually be a “cite” feature to download the citation in a few formats, RIS being one of the most common.

Example: Theory and Craft of Digital Preservation

It can be called by a few names, mostly based on the systems which support it. You might see Ris (Zotero), or EasyBib, Mendeley, ProCite, Reference Manager, and others. But they all follow the same format.

The format is simple plain text format, there are codes which indicate the different field types and tags. The basic structure would look like this:

TY  - BOOK
AU  - Owens, Trevor 
LA  - eng
PB  - Johns Hopkins University Press Baltimore, Maryland
CY  - Baltimore, Maryland
SN  - 9781421426976; 1421426978
PY  - 2018
TI  - The theory and craft of digital preservation
LK  - https://worldcat.org/title/1030899528
ER  -

The first tag always needed to be “TY” and the last tag “ER”. TY stands for Type of reference and ER stands End of reference.

There is actually two versions of the format, this original specification and a later one which added some header information. You can download the full documentation here.

Provider: The name of the information provider (required)
Database: The name of the database (optional)

Tagformat: Name of the tag format used identify fields (optional)
Content: media type for the body of the file (required)

Creation of a PRONOM signature for this text format is pretty straight forward. Looking for the TY and ER string should be enough to ensure the format doesn’t clash with other text based formats. Text formats are notoriously difficult to identify, but when they have expected patterns it makes it a little easier. I had to add a little buffer at the beginning of the signature to allow for the newer header information, but more samples will be needed to see if this is enough to identify the format in all situations. Take a look and see if it works for you!

MP4 & 360

Posted on by Thor Leave a comment

Recently I have been exploring the MP4 format, more specifically the ISO Base Media File Format. It appears to be quite the versatile format. Based on the general Box/Atom format. Don’t mean to go much into the format here as there are so many formats which use this structure, like Quicktime MOV, Jpeg2000, to the more recent Canon RAW CR3. I have also been digging into the DASH MP4 format, but we’ll save that for a later time.

One of the more interesting uses of MP4 lately is 360 or spherical video. They are becoming more and more popular with content creators and also used for mapping like Google street view.

A while back I picked up a Insta360 Nano S camera. It attached directly to my iPhone. With a camera on each side it could capture images and video which could later be processed to produce some interesting results.

Of course it needs to be processed first so it doesn’t look like you are peering out of your peephole. Insta360 provides software for you to process the video into a regular video or some fun creative spherical video that makes you look like you are walking on a small globe.

The formats produced by the Insta360 Nano S are plain old JPG and MP4, but uses the extensions .INSP and .INSV respectively. Neither of which are documented in PRONOM yet. But because of the nature of 360 camera’s there is a little more under the hood. If you would like to look at some samples you can find some here.

The INSP file begins like any other EXIF JPEG file, but ends with a little additional info.

The 360 cameras have some additional information from the different gyros and accelerometers, as well as GPS information. The INSP file stores much of this information after the end of the JPG format. You can also see a string of alphanumeric numbers at the end, which is consistent with most of the files I have seen. One python parser of the additional data calls it the magic number. “8db42d694ccc418790edff439fe026bf” would make a good pattern for a signature.

The INSV files are similar, except they use the MP4 base media format.

Mediainfo indeed sees the file as an MPEG-4 with a AVC codec, but with a invalid extension.

Complete name                            : VID_20210222_170428_005.insv
Format                                   : MPEG-4
Format profile                           : JVT
Codec ID                                 : avc1 (avc1/isom)
File size                                : 41.1 MiB
Duration                                 : 7 s 608 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 45.4 Mb/s
Encoded date                             : UTC 2021-02-22 17:04:18
Tagged date                              : UTC 2021-02-22 17:04:18
IsTruncated                              : Yes
FileExtension_Invalid                    : braw mov mp4 m4v m4a m4b m4p m4r 3ga 3gpa 3gpp 3gp 3gpp2 3g2 k3g jpm jpx mqv ismv isma ismt f4a f4b f4v

In addition to a video and audio track, there is a text track.

Text
ID                                       : 3
Format                                   : Timed Text
Codec ID                                 : text
Duration                                 : 7 s 600 ms
Bit rate mode                            : Constant
Bit rate                                 : 240 b/s
Frame rate                               : 10.000 FPS
Stream size                              : 228 Bytes (0%)
Title                                    : Ambarella EXT
Language                                 : English
Forced                                   : No
Encoded date                             : UTC 2021-02-22 17:04:18
Tagged date                              : UTC 2021-02-22 17:04:18

With a little Exiftool magic, thank you Phil, we can see some of the extra data within the video file.

Serial Number                   : ISS2418ND7XH4H
Model                           : Insta360 Nano S
Firmware                        : v1.17.12.3_build1
Parameters                      : 2 947.866 946.388 964.646 0.000 0.000 90.000 942.993 2891.656 952.520 -0.682 -1.501 89.186 3840 1920 1040
Preview Image                   : (Binary data 578944 bytes, use -b option to extract)
Time Code                       : 62.155
Accelerometer                   : 0.0717358812689781 0.837667405605316 -0.541449248790741
Angular Velocity                : -0.00380666344426572 -0.0143540045246482 0.0170918852090836

Thanks to tools like Exiftool and MediaInfo we can take a peek into some of these formats. New ways of using the existing formats and new formats entirely keep popping up making it hard to know exactly what you have. Initially I just assumed the Insta360 formats didn’t need anything extra as they just used well known format with their own extension, but I needed to look a little closer. Many other cameras are now putting additional data at the end of a standard JPG. It will be interesting to see what new ideas camera developers come up in the coming years.

GoPro has a 360 camera as well and looking at a sample .360 file, I can see it also uses an MP4 base media format, but uses two video tracks to store video from the two cameras. Might need to dig into that format soon as well.

Beef & Babe’s

Posted on by Thor 1 Comment

The 1990’s was a an exciting time for Desktop Publishing. I got my first taste of design in the early 90’s with Aldus PageMaker. QuarkXPress was king in commercial publishing world. For the most part designers and commercial printers used Macintosh computers which QuarkXpress catered to. For those who could not afford the high prices, or used a PC, there was a few options. Microsoft Publisher, TimeWorks Publish It!, and Express Publisher were a few. There was many debates during that time on which software was the best.

I have submitted signatures to PRONOM for many of these:

Express Publisher was proving elusive for finding software and sample files. Express Publisher was developed by Power Up who had been developing a DOS version since the late 1980’s. At one point Power Up decided to sue QuarkXPress for the use of the name XPress. In 1991 Power Up sold all their assets to Spinnaker around the time they released the first Windows version of Express Publisher.

When I first took a look at some samples from the Windows version 1.0 of Express Publisher, the magic header looked familiar.

If it looks familiar to you it is similar to the famous, well nerd famous, JAVA Class file format.

The story goes that James Gosling needed a magic number for his new class format and was in a place they called Cafe Dead when he realized CAFE was a hex value, he soon used CAFEBABE and CAFED00D for his new formats. JAVA was released by Gosling in 1995 for SUN Microsystems.

File Format magic numbers are often used when designing a file to be used with software. Often times it is meant to be a sequence of hex values or a string indicating the file supported by certain software, this is more accurate than the simple extension at the end of most files. They are not required to be there, in fact there are a few formats which are difficult to identify as they don’t use this type of magic number in the header. To learn more read Ross Spencer’s post on magic numbers for digital preservation.

CAFED00Ds and CAFEBABEs

At first I thought it was some sort of homage to the JAVA Class format until I realized the Express Publisher file format was released 3 years earlier. Just a coincidence? I am sure whomever developed this format probably has an interesting story behind it.

These formats are not in PRONOM so lets take a look at what is needed.

The document format for Express Publisher version 1 for Windows 3 uses the EWD extension, as well as EWT for templates. Magic numbers work best for a signature when they are at least 4 bytes long, this gives enough to have little chance of conflicting with another file format. So our PRONOM signature byte sequence would look like this:

  <ByteSequence Reference="BOFoffset">
    <SubSequence MinFragLength="0" Position="1" SubSeqMaxOffset="0" SubSeqMinOffset="0">
      <Sequence>CAFEBEEF</Sequence>
      <DefaultShift>5</DefaultShift>
      <Shift Byte="CA">4</Shift>
      <Shift Byte="FE">3</Shift>
      <Shift Byte="BE">2</Shift>
      <Shift Byte="EF">1</Shift>
    </SubSequence>
  </ByteSequence>

In looking at the earlier DOS versions of Express Publisher they used the extension EPD for a document and EPT for templates. I only have a few samples of version 2 and version 3, but they have different headers.

Version 2 & 3 has consistent bytes starting at offset 4, version 2 using the string PAGES, and version 3 the string EP300. I will have to dig a little more to see if I can find some samples of version 1 to see how they compare and then should be able to submit a PRONOM signature for them.

For the time being, adding “CAFEBEEF” to PRONOM will be a good addition. I wonder if there are any other “CAFE” formats out there, if you know of any, let me know!

UPDATE – There is another format, AnFX Movie, which uses the magic header “CAFEBEEF”. More research is needed to distinguish the two formats.

Hemera Photo-Object

Posted on by Thor Leave a comment

Many years ago I dabbled in a little Graphic Design. Working for a commercial printer in the Pre-Press area, I was very familiar with all things graphics, but never had a great talent for design, especially drawing. I often needed the random clip art for a design I was working on, so I purchased the Hemera, The Big Box of Art, probably from my local CompUSA if that dates me.

Hemera Big Box of Art

The cool thing about clip art from Hemera is it was not your usual JPG or TIFF format, it was in a special Photo-Object format. This format included the raster image, but also included a mask or alpha channel for the main object. They marketed this format as an alternative to the sometimes larger formats of the day. GIF files didn’t have the color depth and PNG was new enough, Hemera was probably hoping this format would be the next greatest thing to happen to clip art.

A Hemera Photo-Object has the extension HPI. Lets take a closer look at a file and see what is under the hood. I pulled this file from Disc 1 on Archive.org

The HPI file has a unique header which should make identification really easy. But what do we see starting at offset 32? A JFIF! Just after a 32 byte header the file has a standard JPG file hidden inside. Now a standard JPG file does not have the ability to support an alpha channel so there must be something else they have within to mask this file. Lets look for the EOF file marker for the JPG format.

Well, well, well. It appears the JPG file is then followed by a standard PNG! Sneaky. The entire HPI file is a 32 byte HPI header, a JPG, followed by a PNG. One could easily carve out each of the formats and save as separate files if needed. There is a script you can use to do this for you, written by Ed Halley. The original Hemera software won’t run on modern systems.

Hemera had a good run for about 10 years before selling off their assets in 2004 to another stock image company. At one point Hemera even purchased the rights to all of Corel’s Premium photo library which I covered in my article about the Kodak PhotoCD format.