Giants Software community manager Lars Malcharek tells us about using Farming Simulator and augmented reality to teach people how to drive a real life combine harvester
When a game and reality meet, extraordinary things can happen. This is evidenced by the collaboration between AGCO and Giants Software, which combines gaming and real agriculture in an unprecedented way. It all began with a simple consideration: How could the new AGCO Ideal Combine Harvester be presented to customers in the clearest possible way? With the help of the game Farming Simulator, was it possible to create a training simulation that came so close to the real conditions that customers could get a feel for the new machine?
The idea, which was initially a rather undirected one, quickly turned into a concrete plan. The Ideal should be replicated as realistically as possible, so that the customers got the feeling to use the real controls. However, they would then not drive the real machine but operate a simulation. The Microsoft HoloLens was supposed to round it off. The purpose of these augmented reality glasses was to provide the driver with all the important details during the simulation. The glasses would, for example, provide the driver with information on the individual Ideal controls when he or she looked at the actual sticks and buttons of the real armrest next to them.
With this idea, the managers of AGCO and Giants Software met to discuss details and see if this plan was feasible. In addition to the CEO of Giants Software, Christian Ammann, Manuel Leithner and Stefan Maurus, two of the gameplay programmers were also on board.
“Of course, we were immediately enthusiastic about the idea”, says Stefan Maurus, one of the developers at Giants Software, "but we also realized that we had many obstacles to overcome if we wanted to make the training simulator as real as possible.” Nicola Zuccolo from AGCO was equally enthusiastic about the collaboration "This simulator will be the perfect way for our customers to learn how to use a new machine”, he explained, "so far customers have only been able to watch a video to learn about the new technology, but with the Ideal simulator they can test all the functions themselves and really understand how the machine works".
AGCO and Giants Software didn't just want to rely on pure testing of the machine, but decided to work on a series of tasks that would be presented to the customer through the game and the HoloLens. By completing these tasks, the customer should be guided step by step through all the new features as it would work in a video game.
“Together with AGCO we planned the complete process of these missions”, says Stefan Maurus, "we wanted to make the simulator usable for all kinds of interested parties. Basically, someone who sat in a machine of this kind for the first time should be able to operate the Ideal through our game. That's why we really started the missions at the base. How to start and drive the Ideal, how the controls work and how to start or stop the header. Then, of course, the whole thing goes deeper and goes into the finer details of the machine."
After the rough planning had been completed, the implementation was already underway. However, it had some problems to overcome.
The first hurdle lay in the technical implementation of the control system. Where gamers were working with mouse and keyboard, or a gamepad, the AGCO Ideal training simulator naturally needed the machine's original side console, but this console was not meant to be connected to a game. There were no connection points to a PC and no approaches how to establish them without any problems.
“We basically had to invent the entire technology,” explains Stefan Maurus, "from electronics to implementation in the game. We also built some electronic parts ourselves and created a physical connection from the armrest with its many control inputs to a PC and established a connection to our software that way." The side console was delivered to the office in Erlangen and work began there. First the hardware had to be created to connect the side console to the PC. Stefan Maurus benefited a lot from his knowledge of electronics to overcome this task.
“I actually come from the field of electronics”, he says, "before I joined Giants Software I worked in this field. So, I quickly had an idea how to connect the side panel to the PC. But there was no real hardware, so we built it ourselves from scratch."
The plan worked out and the side console could finally be accessed from the PC. Now it was a matter of ensuring that the Ideal's data would be transferred as precise as possible in such a way that it would reflect the real user experience of the AGCO Ideal Combine.
The small details were of particular importance. All values transferred from the software to the console had to be as realistic as possible. For this purpose, real conditions up to the GPS signal are simulated and transmitted by the software. Again and again, the simulation was adapted and fed with further data until everyone was satisfied with the result.
Only then did the missions and the fine-tuning work begin. The prepared missions were implemented in the game and connected to the Microsoft HoloLens. Stefan Maurus reports that there were some obstacles to overcome here as well: "The HoloLens has difficulties to recognize exactly what it should. It’s not meant to work at such a short distance. In addition, the buttons on the console are very close together, making it even harder for the glasses to recognize the exact position. So we had to change and rework the software a few times until we had the desired result and the whole thing really worked as hoped".
Then it was done. Almost six months after the idea, the simulator worked and could be presented to the public. The machine was first shown at AGCO's Ideal Combine presentation in September 2017, before the training simulator was presented to the public at Agritechnica, the world's largest agricultural exhibition.
The result was impressive and Nicola Zuccolo also drew a very positive conclusion: "Together with Giants Software we were able to build a training simulator that allows our customers to get to know the machine bit by bit. Our customers are looking forward to trying out Ideal in this simulator and we are very proud of the result.”
Giants Software is also visibly satisfied with the result. Stefan Maurus concludes: "We have learned a lot from this, and when real and virtual agriculture are combined, the possibilities are almost infinite. We already have ideas to explore these possibilities."