OCCRP (Organized Crime and Corruption Reporting Project) is one of the many organizations that were involved with the release of the Panama Papers. Considering the nature of work OCCRP does, they rely heavily on vendor neutral, enterprise grade open source technologies.
We talked to Michał “Rysiek” Wozniak, Chief Information Security Officer at OCCRP to understand more about the organization and its IT infrastructure. Most of us know OOCRP as a platform to support investigative reporting, but Wozniak said that OCCRP has a dual role.
On one hand it focuses on exposing corruption and organized crime, writing about tax evasion and shady deals. The organization was part of the Panama Papers, and Paradise Papers. OCCRP broke the news about the Russian and Azerbaijani Laundromats, and about illegal arms trade between Balkan states and Syria (with the help of Saudi Arabia and the USA) among many more projects. The organization just published the investigation Jan Kuciak (the murdered journalist from Slovakia) was working on.
“The second role of the organization is to support investigative journalism grow in places where it’s most needed. Post-soviet republics, Latin and South America, Africa… we have about 40 partner organizations around the globe,” said Wozniak.
That’s where tools like Nextcloud become critical. “All of this requires cooperation, coordination, and secure communication for hundreds of people,” added Wozniak.
Michał “Rysiek” Wozniak
He joined the organization when Smári McCarthy was setting up a Tech Team at OCCRP. They needed a sysadmin. Wozniak came on a one-month contract, but stayed for 3 years.
The IT services OCCRP offers
Looking at the scale and scope of OCCRP, one can assume the kind of IT services it would offer to users. The prerequisite for any investigative reporting is secure communication. OCCRP offers secure communication tools and Nextcloud is a very important component of the software stack it offers.
“We need to make sure people from places from Bishkek, through Sarajevo, through Johannesburg, to Caracas, can securely exchange information, and source documents.
Secondly, there are tools that help journalists with their investigations, like Investigative Dashboard and ID Search (both of which are created largely in-house and released as FLOSS), and VIS (also created in-house, but not yet open-sourced — working on it though!).
“Finally, there are back-end services more useful for us techies in making sure everything runs smoothly — from server telemetry, through scraper management system (also released as FLOSS), through SSO connecting most of our services, our git repository and CI pipelines, to our own website hosting platform and DDoS protection for those of our partners who would rather focus on journalism, rather than keeping their websites running,” said Wozniak.
Almost all of these components (with just one or two exceptions) are Free Software.
Who is OCCRP serving?
“Our main users are journalists — reporters, editors, researchers, fact-checkers — working on stories. Some of them are our staff, some are regular co-operators or work for our partner centers, some are on an on-and-off basis,” said Wozniak.
This creates certain security challenges not found in other organizations. For instance, there is a spectrum between “being on staff” and “working on a single story”. In other places this division is clearer. This makes figuring access control rules much more challenging..
OCCRP is running a complex stack to manage all these features and functionalities. “We have production and testing servers for our websites, and for our (let’s call it) Platform Services (like Nextcloud). These are all bare metal, dedicated machines. We standardized on docker, and it has proven a good decision — we have also released a number of docker images as FLOSS,” said Wozniak, “We have a several-node ElasticSearch and database cluster, too.”
All these servers are connected by IPsec configuration tool, Metro. So all communication between their servers is encrypted on the IP level. “We also have a number of front-end caching reverse proxies with our special nginx config which we hope to release as FLOSS too,” he said, “a shout-out to good people at Greenhost and 1984.is is in order, we rely on their amazing infrastructure a lot.”
The organization also has dedicated servers that are hosted somewhere else, though.
Being anonymous
Considering the services that OCCRP offer, it’s fair to assume that anonymity of source, privacy and security of information on the site is of extreme importance, one may wonder what measures have they taken to ensure nothing on the platform is compromised.
“Obviously I cannot go into too much detail here, but standard operating procedures involve disk encryption (on servers, and on workstations/laptops/mobile devices), using secure communication tools (all journalists we work with on a regular basis have PGP/GPG set-up, and know how to use it; same for Signal), 2 factor authentication, VPNs, and most importantly — good day to day security practices,” he said.
He added that all journalists receive security trainings, and the Tech Team is there for
them whenever we are needed. If there’s anything even remotely suspicious about an email, for example, journalists tend to let the OCCRP Tech Team know just in case — which makes it possible to catch the actual phishing message every now and then.
“An important part of keeping people safe is making sure security is easy, and that if there’s something suspicious going on, there will be enough red flags for the journalist to catch it. This means, for example, integrating as many of our services as possible with our single sign-on and making sure journalists know that the only link they will ever see in an email about password reset is going to go to this one domain,” said Wozniak. “We are pushing our Member centers to roll-out HTTPS and HSTS on all their sites, and we do this as a matter of policy on everything we host.”
The team at OCCRP is constantly looking at new tools to add to its toolset. “Currently we are testing Briar, and QubesOS, among other things. We are restless to start using Nextcloud audio-video calls,” said Wozniak. “we started testing it as soon as Nextcloud 13 hit the release servers”
They continue to test their infrastructure and services for security issues, and invite others to do so. They have set-up a Responsible Disclosure page with information how to contact them regarding any security problems found in their platform. “We also received great help from YesWeHack, who were nice enough to perform pentesting on two of our services,” said Wozniak.
It’s all bare metal, with a pinch of cloud
The organization relies mainly on bare-metal dedicated servers rented from one of the large EU providers, but it also has certain “cloudy” parts of its infrastructure. “There is an obvious tension about this between sysadmins (who want security and control) and developers (who want to be able to prototype stuff quickly), but I’d say we’re able to navigate this pretty well,” he added.
Heavy user of Nextcloud
Most of the ‘content’ that journalist may be dealing with is files, documents, images or content in other formats. That’s where a reliable, open source and fully secure file sync and share solution becomes critical.
“We use Nextcloud to exchange files within the organization, make device backups, provide upload space for people from outside of the organization (the FileDrop feature is extremely useful!), and soon, hopefully, we will start rolling it out also for audio-video calls,” he said.
The organization has released its own docker image of Nextcloud. “The reason we rolled out our own image is because we wanted to be able to control UID/GID of the php-fpm process (so that we can compartmentalize data on the server better).
Some of the core features of Nextcloud that OCCRP relies on include FileDrop (file and directory sharing between users, and link-sharing; desktop and mobile clients and WebDAV.
They currently have over 100 Nextcloud users, and they hope it will grow.
Thanks to the docker image, they remain on the latest version of Nextcloud (a practice any privacy and security minded entity must adopt)
“Our docker image makes upgrades super-easy — just set the NEXTCLOUD_VERSION envvar to the new version, rebuild, restart, and watch the database upgrades happen automagically. Nextcloud’s sane architecture helps here immensely (this setup would not be possible with many other services),” he said.
There is one feature Wozniak would like to see in Nextcloud. “OpenID Connect (or SAML, but OIDC would be better) integration. This is a big one for us,” he said, “One important part of keeping our users safe is migrating from a dozen services with different credentials, to services connected to SSO where we can better control password quality, 2FA, etc), and easier to fend of phishing attempts (if users know that credentials are *only* handled by a certain domain name, it’s harder for malicious agents to trick them into clicking a weird link). This also means we can better control who has access to which parts of our platform — and being able to manage Nextcloud group membership through our OpenID Connect SSO would be a real boon.
Read more https://nextcloud.com/blog/one-of-the-organizations-behind-panama-papers-uses-nextcloud/
