Welcome back to the re-ignited Technical Team Posts

This is a long-overdue posting, and we will be aiming to do this more regularly as things go forward into 2021! 

There are many exciting plans for the network. We hope that by writing these posts, we can be more transparent with the network's membership about what has happened, what is currently happening, and where we can say what will be happening! We’ll split these posts out into sections where appropriate; such as Infrastructure and Development, to make things a little clearer.

This news article is also available as a blog post at vats.im/techblog-Q420


For a while now VATSIM’s core infrastructure (excluding FSD and AFV) has been in need of a refresh. While reliable, we had a single point of failure and a major OS upgrade due in our future. The current system has been maintained by different members of VATSIM over the years resulting in undocumented configurations in use.

Months leading up to March, the Network Infrastructure worked on reviewing the current state of application source code, databases, websites, and services running to support FSD. The outcome of this work resulted in a plan to migrate VATSIM to a scalable and redundant infrastructure. 

Kubernetes was chosen as our foundation for providing a scalable and redundant compute environment for our websites and services to run in. Every application and service needed to be moved into a Docker container. Some applications just worked, others needed tweaking to run correctly.

With Kubernetes, applications in most cases run on multiple machines providing a highly available and redundant service. When a compute node fails, it automatically recreates the failed node and starts the applications running on the failed node on healthy ones. In cases where applications aren’t able to run on multiple nodes, they will recover automatically very shortly after the failure automatically.

Our database server needed to be upgraded as well. We were on a MySQL version that had just gone end of life. A week or so leading up to March, rolling upgrades were done using Docker to bring the server up to the latest version of MariaDB.

This early database server upgrade and some data schema changes allowed us to move into a MariaDB Galera cluster. This provides a highly available and scalable database cluster for our developers and users.

On March 2nd for roughly 18 hours, we started moving databases and shifting application workloads to our new compute and database cluster deployments. During this time there was no impact to the network for existing users. Within the next few days (the Network Infrastructure team needed rest), most major issues not caught in testing were resolved. Smaller issues resolved over the following days. 

Since then, we have had only one major outage and several minor outages that have gone unnoticed to the wider network due to the redundancies and fail-safes in place. 


Technical Development

vatSys - Jake has been hard at work with vatSys. It continues to progress and we’re seeing more interest in taking up its Eurocat modelled system. 

Euroscope has had several updates on its Beta channel in recent months. Members are encouraged to migrate to this and you can do so at the Euroscope website here and all it requires is to unzip the new version over the top! This removes all aspects of the old voice 
system and adds a variety of other changes which can be viewed in detail here.

The VATSIM Technical Team is in the very early planning stages of upgrading our server architecture. There is no time frame on this and we will share details of this when we absolutely can! 

Something long overdue that we've been getting asked for fairly regularly is a refreshed data feed! If you are using the VATSpy Beta versions, you will already be taking data from this feed!

We'd like to introduce you to Version 3! Moving to production on the 3rd December 0001Z! 

  • Updated more frequently (at present, a minute but we are looking to bring this down to 30, possibly even every 15 seconds!)
  • JSON only for easier parsing by developers - no more string delimited lines!
  • Structured format that allows everyone to focus on what matters most to them
  • Additional data not historically present, including pilot ratings and ATIS codes

The structure for this new format can be viewed at https://api.vatsim.dev/#operation/DataServerV3 

The file is retrievable at https://data.vatsim.net/v3/vatsim-data.json 


Web Development

Application Programming Interface
The VATSIM Tech Team rolled out a new API to replace the cert auto tools divisions were using. This API is available for use at https://api.vatsim.net/api . The new API includes a JSON formatted version of the id_status endpoints, division tools, and more. One of the new endpoints in this new API includes a way to pull user sessions. This means many VATSIM facilities no longer need to store a copy of user sessions that they read from the Data Server, and can simply make a request to the new API. We intend to expand the API in the coming months, to include data such as weather and a wider exposure to other network services. The API will be micro serviced to ensure reliability and speed up development time.  

METAR Update
Many people will have seen, heard or been aware of the recent METAR feed issues whereby they have been jumping back in time. This was due to a change in the feed we were using that meant the file kept jumping back to all hours of the day. We have since moved to a new feed within NOAA that prevents this issue and we have seen a higher reliability on the data since making this change. 

While not focussed overly on the wider membership, anyone who undertakes a pilot rating through an official VATSIM ATO, will find their rating process through PRAMS. This system has been renewed from the ground up to make the ATO’s upgrade and manage members pilot ratings easier and more fluid.

New Statistics Center
In Q3 of 2020, the VATSIM Tech Team began work on a new modern statistics center. The new statistics center will include most of the functionality from the previous site, but will also include broader network statistics. 

myVATSIM was launched back in early August 2020, in line with the release of Microsoft Flight Simulator, and the new VATSIM Pilot Rating system. myVATSIM is now the house for most self-managed membership functions such as email, division and region change. myVATSIM also houses the new Pilot Learning Center. Members in events teams across the network are now sharing all events to this central location! Be sure to check out what's coming up next! 

Retirement Notice - Txt Data File and JSON v1
These versions of the data will be retired on January 1st of 2021. We highly encourage all users of the old text file to migrate to the new JSON feed which can be found in more detail above.

Retirement Notice - SSO
In April of 2020, we announced VATSIM Connect, the replacement for SSO. VATSIM Connect utilizes new and secure technologies such as OAuth2, allowing users to select which data they would like to share. We highly encourage the various facilities and 3rd party organizations which still use SSO to migrate to Connect before March of 2021. SSO will be retired at the end of March of 2021, and will no longer be available for use. From the 3rd December, no new applications and IPs will be considered and all implementations will be required to migrate for further access. You can find documentation for Connect here and you can begin by creating your organization here.


Upcoming Projects

New Website
We are currently working on the new VATSIM.net site. This will be a complete refresh using modernised technologies to showcase the very best of the network. Teasers to come in the future! 

Project Puma
Is all we can say for now! More to be revealed soon!


To Finalize:

Development Discord
If you hold a purely technical way, or are responsible for the technical elements, i.e Webmaster, Infrastructure Manager etc, at an FIR, vACC, Division, Region, Affiliated Virtual Airline, Authorised ATO or other service, and you have not yet joined our VATSIM Technical Development and Web Discord Server, please send us an email at [email protected] to get a link. Here you can communicate with all technical people across the network, share resources, assist others and get quick access to the VATSIM Tech and Development teams!

To finish this off, everyone loves a good stat! 

On average, VATSIM moves between 8 and 10 TB of traffic every month. Roughly equating to ≈ 270GB a day. On Cross the Pond day, we moved 870 GB of traffic. 765 GB of this was through the data server. Levels of traffic this high, over a single 24 hour period have not been seen in VATSIM history! The other 105 GB was traffic routing through SimAware, natTrak and the AFV Map, and other VATSIM services.