Hey guys, Amnesia rebirth steam.
So since the new Rekordbox 6 launched, I've been scrambling to support it for rekordcloud. At first, it seemed that it was not possible since they removed the XML export option and the internal database that Rekordbox uses has been encrypted for a long time. Luckily, I've been able to figure things out and rekordcloud now fully supports Rekordbox 6. 🎉
But this article isn't about that. With Rekordbox 6 a lot of interesting new developments have happened under the hood that I'd like to share with everyone here. I'll go over the new database, the Rekordbox Agent and some other thoughts.
Few things first:
My main operating system is Windows, so all file paths in this article will be for Windows.
%AppData% is a Windows file path shortcut for C:UsersusernameAppData
Rekordbox 5 used a .EDB file to store the database at %AppData%RoamingPioneerrekordboxdatafile.edb. This file appears to be a DeviceSQL database and Pioneer has not made any information about it available. Luckily they allowed us to export our data as an XML file so accessing the database was never much of a priority.
Now with Rekordbox 6, I'm sure you've noticed that it took a while to convert your library from Rekordbox 5 to 6. This is because they've completely changed the database from DeviceSQL to SQLite. The new database now resides at %AppData%RoamingPioneerrekordboxmaster.db. This is an SQLite 3 database encrypted with SQLCipher 4. A bit more about the security below.
Spotify Acquires, Discontinues Serato, Traktor, Rekordbox DJ Softwares By Elisabeth Trebouta On Mar 31, 2019 The new owners of the three popular DJ softwares will discontinue all DJ software, and aims to replace all DJ sets with curated, automixed Spotify playlists by early 2020. Say Goodbye To DJ Software.
My educated guess is that DeviceSQL was too old for them to keep using adequately, especially with the Rekordbox Agent (more below). This was a good opportunity for them to get rid of some legacy baggage. SQLite is a file based database (it doesn't require a server) and one of the most common databases used. Probably the best choice for Rekordbox at this point.
So, the new SQLite database is encrypted which means you can't just use it without the encryption key. Pioneer did this because they prefer that no one outside of Pioneer touches it (there is a forum post by Pulse but I can't find it right now). This is certainly understandable, but the data inside the database is yours so there should definitely be a way to get to it.
Since your data is stored and used locally, we already know that the key must (at some point) be present on our machine. Usually it's hidden in an executable file (possibly obfuscated, this means the key is garbled on purpose so it's hard to use without understanding the code around it). Knowing the key must be local somewhere, gives good hope that you can find it. Although sometimes this can be really hard.
You're probably wondering what the encryption key is. I'm not sure it's wise for me to just post it here. I'm not sure how upset Pioneer would be with me so I prefer not to. But if you read this article thoroughly and poke around in the source code, I'm sure you'll find it.
I thought the key might be license or machine dependent, but it appears that all databases are encrypted with the same key.
The other real interesting addition is the Rekordbox Agent. This looks like what they're using the achieve the new cloud sync feature. It's a completely separate program from Rekordbox 6 and it launches when you launch Rekordbox 6. Even if you don't have cloud sync, it still launches.
The Rekordbox Agent is an Electron app. [Note: removed some detailed reverse engineering instructions here.] After looking inside the source, the app appears to run the Express web framework for Node.js. We can combine this information with what we see in the Rekordbox Agent log file at %AppData%RoamingPioneerrekordboxAgentlog.log. It contains lines like these:
Rekordbox Agent launches an Express instance so it can listen to incoming events. By default it uses port 30001. You appear to be able change this in %AppData%RoamingPioneerrekordbox6rekordbox3.settings by changing the CloudAgentPortNumber key value.
There are lots of routes here and I haven't explored all of them yet, but it's fairly safe to say that the Rekordbox Agent is in charge of up- and downloading your tracks and database. There are references in the source code to copying files (eg controllers/file_copy_worker.js looks like a multi-threaded worker). The source code also contains the database model for every table (in the models folder).
Other interesting things I noticed in the source code:
- Looks like your cloud sync backups go to Amazon AWS S3. More specifically (in my case), the rb-cloud-data-eu bucket in region eu-west-1. I'm in the EU so it might use multiple regions, depending on your location. But since I see the eu-west-1 reference more often, I think they only use one location. You can actually see this yourself in the Rekordbox Agent log file.
- There are references to Google Drive and OneDrive together with Dropbox, so they may support that in the future. This is paired with something called rekordbox cloud (hopefully not to be confused with rekordcloud in the future). Maybe they are just using Dropbox now as a quick fix (and let you pay for it) and later they might include their own storage solution?
- There is a reference to a Spotify file type. Maybe this is coming in the future? Take this with a grain of salt though, it was one small reference that did not occur elsewhere.
Extracting Your Data
The XML used to contain (almost) all our data, with a few notable exceptions such as MyTags. Now that this is no longer an option, we'll have to do it ourselves. Luckily, the SQLite database is clearly structured so getting our data out of there is no problem. I'm not going into details here, because it's really not very interesting how the database is structured.
There is one problem here though. The beatgrids are not found in the database. They appear to only be located in the ANLZ0000.DAT files. These are binary files that contain the exact beat locations (and more information such as cue points, waveform and more). These files are found at (for example) %AppData%RoamingPioneerrekordboxsharePIONEERUSBANLZ00cfc4d7-802f-430d-a845-0bd9a87cb09c. Looks familiar? These are also on your USB stick used by Pioneer CDJs. These files contain your pre-calculated waveforms, cue points, beatgrids and any other information the CDJs need.
The database points at the right .DAT file per track so finding out which file has the right beatgrid is easy. Getting them out of the .DAT file is a bit harder, but thankfully has been thoroughly researched and explained here. This certainly saved me a lot of time!
Rekordbox 6 can still import XML files. Honestly, I wonder why since the move to remove XML export is a step closer to a walled garden approach. But since you can still import XML, I've decided to use that method (for now) to import your data from rekordcloud back into Rekordbox. It is definitely possible to write data directly to the SQLite database, but since I don't have any documentation about it, it's safer not to do this yet. There might be unforeseen ramifications that could cause data loss.
One pain point of the XML export was always that it did not include MyTags. With access to the Rekordbox database, we can read now (and write) MyTags. This opens up many new possibilities in managing your MyTags. Thoughts about this? Let me know!
Maybe there are novel ways to use the full information of the database that wasn't possible earlier with the XML export. Discuss below! :)
Preface: I'm not a lawyer or legal expert by any means so take everything I say with a grain of salt.
Rekordbox encrypts the database that contains your own data. This is unlike any of the competitors. Traktor and VirtualDJ both have XML files that you can look into with any text editor. Serato saves most data into ID3 tags but can still be viewed easily with a tool like kid3. Until now this was no big deal since the XML export provided most data.
If you're in the EU, you fall under the GDPR. Under Article 15 of the GDPR, it is stated that you have rights to access your own data. Pioneer must provide a way to do this. The XML export was a suitable method but no longer exists. There is no automatic way to provide access to your own data anymore. We'll see where this goes but I don't think they have a choice to either bring XML export back or open up the new database.
If you're in the US (or elsewhere): I don't think there are any applicable privacy laws here. Convince your representatives to take privacy more seriously.
Edit: most likely this does not fall under the GDPR since it is not 'personal data' (see definition in Article 4.1).
It could be different
In case Pioneer reads this: do the right thing and open up Rekordbox.
Bringing back the XML export is an option but if Pioneer wants to do it right:
- Remove the database encryption
- Document the database
- Support third party developers
An open platform leads to innovation. Closing it up will cause Rekordbox to stagnate. Opening it up can lead to a whole ecosystem around Rekordbox with limitless possibilities and integrations.
Thoughts or corrections?
If you have any thoughts, or notice a mistake, let me know below.Follow rekordcloudFollow rekordcloud
You can download your library at any time by pressing the
Download button in the top of the screen when you're logged in.
You can also download your library in the
Download Library menu and download either a Rekordbox, Traktor or VirtualDJ version of your library.
To learn how to import your downloaded file back into Rekordbox, Traktor or VirtualDJ: read on below.
Serato uses the Companion App to write file tags.
Rekordbox 5 & 6
Importing a Rekordbox XML file is a bit complex, but just follow the steps below. These steps are the same for Rekordbox 5 and Rekordbox 6. Rekordbox 6 is fully supported.
First we have to set the Rekordbox XML location in the advanced preferences menu. Only after that will the XML menu appear.
- Make sure you didn't change anything in Rekordbox since you last uploaded your library since those changes may be overwritten.
- Download the Rekordbox XML file
- Open the Rekordbox preferences in the File menu
- Go to the Advanced tab and set the
Imported Librarylocation to the XML file you just downloaded.
Rekordbox Download For Windows 10
Make sure that 'Rekordbox xml' is checked on the View tab under Layout.
Close the preferences menu and restart Rekordbox.
You will now see the new
rekordbox xmloption in the menu. Your downloaded XML should already be loaded but if it isn't, just press the tiny Reload button. If you don't see the XML tab, make sure you restarted Rekordbox.
- The easiest thing you can do now is to right click your top most playlist called
Import Playlist. This imports all playlists and all tracks inside any of your playlists. If you have tracks that aren't in any playlist then you may need to import the entire collection too. Importing the entire collection may take a while so it might be faster to only import specific playlists.
- Warning: The latest versions of Rekordbox have a bug that prevents importing of XML files. Until Pioneer fixes this, you should use version 5.6.0 to import tracks. This bug also exists in Rekordbox 6. More info and the download links to the older Rekordbox can be found here
To import an NML file back into Traktor follow these steps:
- Right click
Track Collectionand select
Import another collection
- Find the NML file you just downloaded from rekordcloud.
- Traktor will ask you if you want to keep your file tags or use the tags from the collection you're importing. Select collection here.
Sometimes playlists don't show up immediately, restart Traktor if that happens.
Your tracks and playlists are now updated and you can start using them.
If you relocated or deleted any files: you should right click a track in your collection and select
Check Consistency. This lets confirm relocated tracks are found and allows you to easily remove deleted tracks from your collection.
Ddj 400 Spotify
You should have downloaded a
ZIP file. This file contains your
database.xml with all your track information and a
Importing a VirtualDJ library is just replacing these files.
- Make sure VirtualDJ has the following settings:
If your music is on an external/network drive, delete the
database.xmlfile inside the
VirtualDJfolder on that drive.
Go to your main VirtualDJ folder:
Playlistsfolder and the
Copy the contents of your downloaded
ZIPfile into your VirtualDJ folder. You should have a
database.xmlfile and a
Playlistsfolder in the same place again.
Start VirtualDJ and your library is updated. You will find your playlists under
Lists & Advice->
Can You Use Rekordbox With Spotify
VirtualDJ doesn't export empty playlist folders. So if you had empty playlist folders that are now missing, it's because VirtualDJ never included them in your database backup.
If you notice that your tracks aren't getting updated, it may be because you did not delete the
database.xml file on your external drive. This file contains old information and VirtualDJ takes priority on this file. After deleting and importing it, VirtualDJ will recreate it again with the new track information.
Serato uses the Rekordcloud Companion App to write the tags of your music files and to update your crates. You can download the Companion App here
After downloading your library with the Companion App, just open up Serato and you're done.