From October 16th to November 6th we ran a public beta test campaign for the v1 of our AI Legal Assistant. During this test, we held several campaigns in order to gamify and encourage our community members to get involved. In this article we provide some insights, metrics, and results of this nearly month long campaign!
The V1 applications within the AI Legal Assistant platform that were tested include the AI Assistant (Contract Creation), the Contract Reader, zk-SNARK certificates, and the verification of Certificates.
We must also take this time to thank absolutely every single individual who made this beta test a success! From our community, to our campaign participants, to our testers!
Background
Co-organized with Netizen across several platforms, we aimed to reward and acknowledge those who joined the campaign, tested the product, AND spread the word about our AI technology. We rewarded BOTH our best beta testers and our community engagers in this campaign in order to drive both awareness and action.
Participants had the chance to win $5000 worth of $VAI tokens over the course of our 8 mini-campaigns within the overall beta test. There were multiple ways to engage, from creating compelling content such as memes, videos, infographics, tutorials, or articles, to amplifying our messaging on social media. There was XP awarded and leaderboards made by Netizen in order to track players directly in the VAIOT Telegram community.
Participants, of course, could also interact directly with the AI Assistant and get directly involved in it’s development by reporting bugs and development suggestions using our Jira form. The Jira testers who were doing this heavy lifting had a separate leaderboard hosted by VAIOT.
Despite several set backs and delays in the beginning, we witnessed outstanding results across the board. We believe this was done thanks to not just the hard work of the developers and VAIOT team but the patient and driven beta testing community we built along with Netizen.
If you would like to read more about how these leaderboards, campaigns, and further details regarding the test, please read the introductory article to the beta test here.
Additionally, you can also read the update article for the second week of the test, which introduced zk-SNARKs into the beta.
Beta Test Results Overview
Test Data
🔹954 Reports on Jira
- 172 bug reports
- 782 development suggestions
- 42% of those reports were considered helpful in development.
- 35 of the bug reports were considered immediately critical to function and have or are being addressed in development sprints.
🔹2.54 million non-unique API calls from user interaction
🔹90,000 interaction on assistant.vaiot.ai
🔹2097 unique users tested the AI Legal Assistant v1 BETA
🔹425 unique wallets connected to AI Legal Assistant
🔹53 zk-SNARK certificates were created.
🔹577 contracts were created within the AI Legal Assistant.
Engagement Data
🔹Twitter — our engagement essentially doubled from usual.
- 438k Impressions
- 1200+ link clicks
- 9000+ likes
- 2000+ retweets.
🔹Telegram chat gained 1500+ new members, sending group membership into 5 digits.
🔹Medium — our engagement doubled here as well.
- 2300+ reads
- 30+ new followers
Campaign Data
🔹Netizen RAID
- 2,700+ pieces of content created
- 400+ user referrals to new users
- 349000+ VAIOT leaderboard XP earned.
🔹Jira Beta Test — 20+ active beta testers — 5 winners
🔹TaskOn — 2200+ participants — 24 winners
🔹Galxe — 300+ participants — 12 winners
🔹Zealy— 1000+ particpants — 24 winners
Sprint 1 Winners
@medicixl
@BNXN17700
@x3350
@FBid2016
@Seylin2021
@NYSC2022
@Ignatio2008
@0xpzgynn
@mubbystar01
@arnievan762
@Zephyrakkk
@JimboArr
Sprint 2 Winners
@Green133695
@ChihuahuArt_
@GreedOVG
@lautexy
@ellasmith05623
@whytboi25
@Fries1415786
@Ezinne000582594
@Helen9289915675
@Bestbuddy00
@influence4
@M4ykke
AI Legal Assistant Beta Test Changelog
Throughout the beta test, there were many changes made based off of the experience of our testers. This list is mainly exhaustive, but does not include changes that will still be implemented after the writing of this article.
Word from the Developers
“35 mostly bugs were immediately entered into the backlog. As those requiring intervention as urgent.
These are non-duplicating cases. So we can safely say that half of the bugs were bugs that seriously impacted the operation of the application.
In my opinion and that of the developers, the tests were very successful and pointed to many problems that would not have been detected with the previous scale of (internal) testing, because they did not occur often, making it difficult to find them during internal testing.
What’s nice is that the vast majority of bugs were described properly. That is, it allowed errors to be replicated. The reports contained either videos or a description of the error along with points for replicating it, or screenshots of the console allowing the error to be identified in a shorter time.
There were also a lot of suggestions, although here they were very different. I am glad that from such quantities it was possible to choose not only the good ones, but also pay attention to their repeatability, which increases their priority for implementation.
As a good example, I can give you a button that allows you to go back in a dialogue. This option is currently hidden under the “back” or “go back to…” command, which we talk about in the tutorial at the very beginning, but you can see that this option does not work as a stand-alone solution and can only be an accompanying option.
Thank you all for participating! We couldnt have done this without you”
10/18/23
While the ability to create certificates based on zk-SNARKs will be temporarily disabled, but rest assured that the core functionality of our AI Legal Assistant remains intact.
Here’s a summary of the adjustments made:
- Users can view a list of their created contracts within the Contract Reader, mirroring the full version.
- They can also download a full draft of the contract (the same one they received via email during contract creation) while connected to the appropriate Crypto wallet. Connecting a wallet allows you to access the list of contracts connected to a given wallet. When saving the contract as a smart contract, the user must connect the wallet. Similarly, if you want to see the list of contracts created and saved as a smart contract in Contract Reader, you must connect this wallet. This is a form of user verification. Without a wallet, the user will not be able to use Contract Reader.
- The “Buy Certificate” button will be disabled temporarily to prevent users from engaging in a non-functional feature.
- The “VerifyProof” application remains accessible, allowing users to explore its functionalities. However, full utilization necessitates pasting a downloaded JSON file, a step currently hindered by the certificate creation process delay.
- Other verification options, such as QR codes or direct verification of proof in a smart contract, will also be unavailable due to the existing challenges in creating proof.
10/23
- Added a timeout to the payment. Due to the use of a test network and test tokens, sometimes the transaction did not go through and the CLU responsible for communication gets lost. Now, if the transaction does not take place within the specified time, CLU will consider the transaction unsuccessful. and it will show communicate -FAILED payment. The transaction will be able to be repeated.
10/24
- Added zk-SNARK certificates into the live Assistant so that testers could navigate the entire Assistant path.
10/27
- Improved QR Code connect efficiency
- Optimized AI. For example: errors related to synonyms (they work in some places, others don’t) and response time optimization.
10/31
- Fixed both frontend and backend bugs. For backend, those related to CLU and AI feedback, but also the path itself, validations, etc. On the frontend, errors related to connecting the wallet or reading text by the Assistant.
11/7
- Fixed the “Start again” button. From now on, this is a reset of the entire conversation. There will no longer be a loop with the message: Hi, my name is VAIOT. How may I help you?
- Improved wallet switching. We have updated web3-react/core to the latest version. This will allow you to switch between Metamask and Coinbase more efficiently and error-free.
- Removed Portis from the list of possibilities. It has been withdrawn by the publishers. Due to the small number of users and the large number of errors generated. We will replace Portis with a new tool in the future.
- Improved the security of the application. Which will improve the stability and smoothness of the application (we have significantly reduced the phenomenon of hanging and freezing of the assistant)
- Added changes to field validation, e.g. adding abbreviations as country names (here we use the official FIBS database), where it should not be possible, negative numbers cannot be entered.
- Eliminated the error of sending the contract to the e-mail address provided for the previous contract.
There are also many minor changes that were introduced based on reports during testing.
11/10
- Updated user/dialogue state handling.
- Added FIPS codes recognition in country name prompt.
- Added stricter validation rules.
- Fixed a bug where new mail provided by the user was ignored and the agreement was sent to the old email.
- Optimized assistant prompts loading logic.
- Optimized our cloud gateway performance by using virtual networks.
- Added more security checks to help us counter DDoS attacks.
- Migration from WalletConnect V1 to WalletConnect V2
- Upgraded logic for handling other wallets like Metamask and Coinbase Wallet
- Ability to restart the conversation at any time
- Improved logic of creating new conversations, making it more secure and reliable
- Updated all packages to the newest versions
11/13
- All changes implemented into the Assistant and relaunched into the production environment.
About VAIOT
VAIOT offers a portfolio of blockchain-based AI Assistants for businesses and consumers to provide automated services and transactions. Faster, easier, and affordable.
For more information about VAIOT, visit our website www.vaiot.ai, or have a look at the Twitter and Telegram Community.
