Challenges of Location Based AR Development
Author: Matt Stentiford
Senior Developer at Pretia, who's been in the industry for over 15 years. With more than 10 years of experience developing AR applications, exhibits and experiences. Current passion is looking for ways of creating new and immersive experiences in XR.
Overview
Over the past few years, AR technology has rapidly been improving, from basic image tracking, to markerless AR and plane detection, and most recently, accurate mapping of a device to real world locations via VPS solutions such as Pretia’s own SDK.
When creating location based AR Applications, there are a number of unique challenges that developers face, especially when compared to traditional app development workflows. In this article, we will cover some of these key differences, focusing on the following three processes:
Location Mapping: Thoughtful and thorough mapping of locations is critical.
When creating location based AR apps, one of the first steps that is important to tackle early on in development, is ensuring the creation of robust maps that can be used for relocalization.
Content Authoring: Location based AR scenes require the content to match the real world environment.
Without the right tools, this can be challenging and time consuming to ensure the AR content is correctly positioned and fits believably into the environment.
Testing: Location based AR Apps can be hard to test.
The final app will have activations that are tied to specific locations. This can make it challenging to test or debug without creating a new build and heading on site to a location, which may not yet even be accessible. Many AR frameworks have limited support for simulation or “remote” testing from PC. For example ARFoundation still does not yet have any official simulation in editor or remote (Although simulation at least is something that Unity plan to address in ARFoundation 5.0.0).
Location Mapping
Selecting and capturing a good map is perhaps one of the most critical steps in order to create a satisfying location based AR experience. Some of the most important factors which will influence mapping include:
Weather conditions
Rain / snow / harsh sun / shadows all make relocalization very difficult.
Time of day / lighting / shadows
Season / time of year
Moving / changing environments
Locations with vegetation, people, furniture that might often be moved (such as chairs etc) can all be problematic.
Picking indoor or undercover locations can greatly reduce the impact of many of these potential issues. If possible, map and test with final lighting conditions as soon as possible. Exhibits often leave lighting until near the end, however it is best to try finalize any lighting adjustments before mapping and testing, as this can greatly impact the speed and accuracy of relocalization.
Relocalization:
If the location changes, or the map needs re-creating then this can lead to many difficulties in re aligning the content with the new map. Try to ensure that you pick locations that are unlikely to change over time.
When using the Pretia SDK, the Scanner app provides an easy to use option for testing relocalization. Be sure to perform a few relocalizations after creating your map to ensure that it relocalizes easily. You may also want to test at different times of day, or different lighting conditions that your users may experience as well, in order to ensure the map acts as a reliable relocalization target.
For further reading on creating a good map, then be sure to have a read of the documentation here:
Game Design Considerations
Selection of locations and mapping should always go hand in hand with the game or application design. Some AR experiences will need more accurate placement than others, and there may also be ways of limiting the AR experience so that users will only activate it at a specific time of day, or under specific lighting conditions.
Some things to keep in mind when designing location based AR experiences:
Accuracy:
Relocalization using the Pretia SDK will currently have a pose variation of generally around 0.1-0.2m between activations. Therefore you should try to design experiences that do not rely on centimetre accuracy in the content placement.
Content should be placed in open spaces, or on planar surfaces. Avoid trying to match AR content to thin, non planar surfaces.
Limit the play sessions
It is a good idea to limit AR experiences to similar times of day to when the place was mapped for outdoor experiences if possible, so as to ensure the user can relocalize quickly and accurately.
If the user is likely to play at different times of day, then scan multiple maps at different times of day in order to ensure the best relocalization experience.
AR is a very resource intensive process, so it is best to design experiences that can be enjoyed in 1-2 minute play sessions, to preserve users battery life and prevent the device from overheating.
Content Authoring
Content Authoring is the process of placing content relative to mapped locations, or anchors that have been placed as part of the scanning process. This process can often be challenging, ensuring that content is correctly aligned with surfaces, and positioned so that there is minimal occlusion from other objects in the environment.
Pretia provides a simple yet flexible content authoring system within the Mapping App. After mapping a location, content may be placed in AR, and then imported into the Unity Editor. You may either use preset prefabs as references for placement of final content, or you can upload models for direct placement of final game assets inside the content authoring tool itself.
With thoughtful use of the content authoring tool, developers are able to save a lot of time attempting to align content correctly to the environment.
For further information on the Content Authoring tool provided as part of the mapping app, feel free to refer to the following documentation:
Tips For Content Authoring
Create placeholders for your content as early as possible, so that they can be added during the mapping process, and provide a guideline for how the final assets will fit into the space.
If you do not have assets ahead of time, then placing the default simple primitives in the Content Authoring tool can still be quite a good reference in terms of the bounds of the objects to be placed later. Prefabs can then be created using these bounding boxes, and replaced at a later date when the final assets are available.
Author your content on the best device you have available.
Different devices have different capabilities. While it is possible to author content on almost any device with basic AR capabilities, you want to have the most accurate relocalization possible before you start placing content, as well as access to the Depth API for depth occlusion, preferably with a device that has a ToF, or lidar sensor for accurate depth data.
Make use of depth, meshing or detected planes in order to ensure objects are placed correctly.
Ensuring that objects are placed correctly on surfaces, and not under the floor, or behind a wall is one of the most important things in ensuring that content feels like it fits the environment, and does not appear to float or drift when the user views it in AR.
Add occlusion meshes and shadow planes where appropriate.
Pretia’s Content Authoring system provides an easy to use way of placing simple occluders and shadow surfaces. Place these for objects such as real world walls that may occlude your AR content.
Although newer devices are capable of realtime depth occlusion, and detected planes can be used as shadow surfaces, manual placement of these surfaces can provide a much more reliable and convincing experience.
Stay tuned for a further guide which will go into more detail on occluders, and how you can use them in order to improve the realism of your AR experiences.
Testing
When performing traditional application debugging and testing, there are a wealth of tools available for the task. Engines such as Unity provide an amazing workflow for testing in Editor, and having near instant iteration from making scene and script changes, to testing in play mode, debugging and stepping through code as it executes and so on.
Unfortunately, AR frameworks today do not provide many tools to help in this regard. Although there are third party plugins for remote testing (for example AR Foundation Remote), or some simulators (such as the Unity MARS device simulator). These tools are still not as reliable or easy to use as some of the options available for standard mobile app development.
On top of these shortcomings, location based AR Apps have the added challenge of being linked to a real world location, which must be scanned in order for the experience to activate. Content must be accurately placed in order to match the environment, which is sometimes difficult or even impossible to access in advance of the release of the App.
The Pretia SDK provides a number of solutions and guidelines for dealing with these issues:
The Mapping App has a built in content authoring system, explained earlier, for quickly prototyping a scene as soon as you are able to access and scan a location. Relocalization can be tested within the mapping app, and assets can even be uploaded for use within the app, in order to rapidly view how they fit in the environment.
The online relocalization service allows quick and easy swapping between different maps. allowing developers to test in one location and use a different location for release builds.
The recording and playback system which is currently in development should also prove to be a powerful tool to allow testing in editor, with recorded camera and sensor data. This allows developers to reproduce the in app experience while still being able to use traditional editing and debugging tools and rapid iteration that are possible on PC.
In combination, these three elements can allow for a variety of workflows depending on the circumstances, for example an app can be developed and tested in one location, then mapped and adjusted in a matter of minutes by a team on site in the final location. In practice it is always best to allow plenty of time for testing and iteration before an event, however sometimes this may not be possible. We provide the tools to make the workflow as flexible as possible.
Conclusion
Location based AR is still in its infancy, and Pretia is constantly working to improve the tools, workflows and guidelines for developing these kinds of applications. Hopefully some of the above points give you some thoughts about what to consider while developing your applications, and we look forward to seeing all the amazing Location based AR experiences that we know developers around the world will develop in the future!
Try our AR Cloud Platform "Pretia"?
We are hiring engineers!
この記事が気に入ったらサポートをしてみませんか？