This text was generated using AI and might contain mistakes. Found a mistake? Edit at GitHub

Episode 293 - Collaborative Software Architecting with LLMs with Claudine Allen

Welcome to a new short episode of Software-Architektur im Stream.

We are here live from the ISAQB Software Architecture Gathering in Berlin.

Here with me is Claudine Allen from Jamaica.

Claudine, will you introduce yourself a little bit?

Hi, my name is Claudine Allen.

I’m a lecturer at the University of the West Indies, MONO in Jamaica.

I’m also an ISAQB board member.

I work primarily with the university working group and I’m also a CPSA foundational level trainer in Jamaica.

The talk you’ve given last day was about how to use AI with software architecture to create a software architecture.

Can you tell a little bit about your approach, how you use AI?

The main thing I’ve been trying is looking at, for example, the way that we approach software architecture.

It’s sort of methodological, so we do have different steps that we use usually and different aspects that are important.

So first of all, clarification of requirements, understanding your constraints and so on.

And what I try to do is incorporate LLMs in a way that they help me to do background research faster when it comes down to, for example, standard documentation.

It helps me to answer questions in terms of information that I need faster.

Furthermore, I find it very useful as long as you have the appropriate contextual information in terms of rewriting and clarifying your requirements.

For example, having your quality scenarios well written and very clear.

So I find it useful in that way.

It’s also useful, again, as you go about architectural design, if you’re using an incremental approach and doing small parts at a time, providing the necessary context as you try to solve small problems.

So, for example, once you have decided on, OK, these are the primary blocks that you need for your static structure.

And you can use the LLM to help to define those structures.

For each structure, you can get information.

You can get help, for example, with the logics of the structure or what the interface might look like, things like that.

So I would not try to use and I wouldn’t recommend using an LLM like ChatGPT or Software Architect 4.0 to solve the overall architectural problem at once.

But a stepwise approach that’s incremental, I find it to be quite useful, especially if you’re looking at the LLM as kind of an assistant, you know, asking it questions.

Once you get responses, trying to clarify, doing your research, yes.

So you use it more for, I mean, there are many people out there who try to use AI to create the whole architecture.

But as I do understand, you advise people to use it for brainstorming, for research, to get deeper inside inspiration.

And so this works when you work on a topic which is well known to the LLM, right?

Yes, yes.

Actually, that’s very important.

The LLM is, of course, depending on documents that it has access to on the Internet.

So when the topic is not something, so there may be a green field of projects where you might not have that type of information.

Sorry, novel types of projects in novel domains where the information might not be there.

So the type of help that you can get will be different.

But information, for example, on methodologies is available.

So methodologies like R42 or 4 plus 1 views, that information is available.

So it can help you in a methodological way.

So it can help you with identifying a template that’s useful for, let’s say, an interface definition.

So you work a lot with students, and so you not only use AI for research in the business domain, but also for how do I write an ADR and things like that?

That’s correct, yes.

That’s a great idea.

The topic you presented was, so the business domain was about sign language.

Yes.

I think it’s quite interesting.

Yes.

So can you elaborate a little bit on this, how AI helps with this domain?

I mean, it’s a special domain which is not really known to most of us.

Right.

And I wouldn’t know how I should approach this.

So AI also helps there?

So in the experience that I had, there were things that I did not know in terms of how do you translate natural language into something like sign language.

Sign language is not necessarily a one-to-one.

I know some sign language.

I used to do sign language.

So you don’t have necessarily a one-to-one relationship between words that people say and a specific sign.

Sometimes there’s a sign for a whole expression.

So I kind of wondered how would this work.

And the research that was available online, I was able to access that.

Obviously, you can go ahead and read the research papers.

But when I used the LLM, it pointed me to the right papers, and it answered some of the questions that I had very quickly.

So for example, the idea of something called glossing, which I’d never heard before.

What’s it?

So it’s basically, it’s kind of like an activity between getting the natural language, the audio, into text and then going from text to sign language.

So glossing is like taking the text and translating it into a version of the text or a nuanced representation that matches more with sign language.

So it’s in between the two things.

Okay.

That, for me, is a quite interesting part because when you talk about sign language and translation of audio, I first thought, hey, if you have the text, why need the sign language?

But I had to learn that this is a whole different thing for people who need the sign language.

And not only for sign language, there are, even in natural language, there are languages that don’t have necessarily the same kind of structure as, like the structure of languages are different.

So there are nuances between two languages.

And what I understand is that that glossing helps, that glossing process helps to, I guess, smooth out the translation from one language to another.

Okay.

So in this case, with this special architecture, with a domain which is not well known to everybody, the AI helps with brainstorming about the architecture, how to build it, but also about the business domain.

Exactly.

What glossing is and how to build a pipeline for it.

Yes.

And especially the issue of a part of translation is not just the words.

Translation also includes the way that people, the gesticulation and so on.

So one of the things that I realized is that the avatar, there are two different ways that you can process this, is you can take the gestures and expressions from the human, because the purpose of the translation is to take the human, the beta, and translate what they’re saying into sign language.

So you can either capture the expressions, the facial expressions, the hand movements, the body movements, and so on.

You can capture that and replay it in the form of the avatar, or you can ensure that the translation from the avatar, depending on the kind of resources you have in terms of time and so on for development, if you can keep the timing of the avatar close enough to the visuals of the actual speaker, then you wouldn’t lose the gestures because then the person who’s viewing would see the sign language along with the person’s gestures.

So there are two approaches that could be used depending on the kind of resources and time available.

So I remember there was a quality criteria that within two seconds the translation should be there.

From my understanding, it doesn’t translate word by word, so it needs to know the full sentence, and after it got the full sentence, two seconds.

So the whole process, what I wanted to accomplish, or what the quality requirement is describing, is from the time that the phrase that’s being translated is said to the time that the avatar completes the translation, then we don’t want more than two seconds to elapse because we want the two things to be.

So the approach that I am taking is to keep the actual speaker beside the avatar so that the sign language can be interpreted along with the gestures of the original speaker.

Okay, and you also said that the AI also helps with those quality requirements.

Yes To to find them or to formulate them both both so it might not be able to find the quality requirement from scratch But the research that is necessary to so for every quality requirement Basically, it’s work.

It exists within a domain So if I’m gonna talk about performance, I need to understand how do people measure the performance in in relation to You know any particular thing if I’m gonna talk about availability.

I need to have the correct vocabulary I need to know the possible metrics if I’m gonna talk about usability There are a bunch of different ways that we can measure usability and describe it.

So for Novice architect and ID with students and I’m mainly in academia.

So there’s a way in which There’s a way in which You you know not being in the industry all the time there are so many things you want to learn so many things you want to try and experiment with and what I have found is that being able to describe The response and a response measure for a quality requirement the LNM.

Yeah, I can help you with that So you may be able to explain what it is that you want or what is important in this quality in English But not necessarily with a precision necessary for architecture work and LLM can help with that Okay now I remember that the architecture was more or less a pipeline a processing pipeline and now when it comes to to all those single steps to turn audio into Transcription, so I guess you use some Not necessarily a large language model, but machine learning I guess so so Not really Yes, there’s some machine learning that will be necessary because you do have to you’re collecting a lot of data about various signs And so yes, there’s there be some machine learning involved.

So I guess there are already libraries away Also for the for the glossing All those things there are libraries and the good thing was that LLM was able to recommend various technologies for all the stages Because it has a knowledge from it access to all this documentation so Did you mainly use the knowledge from the training data or did you use a web research?

Which is mainly research because we haven’t actually trained a model or anything like that yet.

It’s really just a matter of The starting point of developing this architecture Okay, that’s great you mentioned arc42 and I was happy to hear that arc42 is used in Jamaica Jamaica Jamaica we talked about the pronunciation pronunciation so that’s Jamaica, yes Can you elaborate a little bit on that?

Because I think it’s quite fascinating to see how arc42 is used.

Okay in the world so arc42 is new in a way to Jamaica as is the SAQB and the training and so on over the last three years.

We’ve been training students Using the CPSC curriculum now Maybe about five six years ago No, no, no, no, not so much about four years ago.

We started software architecture as a course in my university a lot of what we did and the way the curriculum was developed was based on based on Textbooks by people like Len Bass and Rick Caseman and so on and we used a lot of the tools and Methodologies from the Software Engineering Institute, which I found just amazing In that process I started being involved with Excel going to Excel conferences and so on and that’s how I found out about the SAQB Once I found out about the SAQB I did training with the SAQB I did the CPSC and what I found was so one of the things I learned was well The experience with SEI was phenomenal But I did realize that a lot of the work that SEI did was for really heavy mission critical systems Because you know work with the Department of Defense and government and so on so a lot of times the methodologies were not necessarily suitable for the types of the types of industries that the students went into But the principles were always relevant and so we’d have to tweak things to match these smaller projects When I got introduced to arc42 and I saw the methodology and the different views the four simple views and the way that arc42 documentation was set up I decided to start incorporating that into our course and at the same time the students were being exposed to the CPSC of training as well But once I started doing that the difficulty level for documentation went down significantly for the students arc42 presents Exactly what they needed for the types of business projects that they were more likely to be solving it’s easier for them to understand the Learning curve was a lot less steep and I didn’t have to spend quite as much time clarifying what different sections meant So that was a good transition for us That’s great to hear.

So my experience is that LLM the Gen AI already knows about arc42 because it’s Open-source it exists for now 20 years.

It’s in the training data In what format do you Document the architecture.

Do you use a DOCSIS code approach?

Markdown or do you use Word or wiki?

Right.

No, we just use Word or I might use a latex template for it But for right now the students are using Microsoft Word Okay, and as I understood the students writes a documentation with the help of AI not AI Right.

Absolutely.

So not only in terms of writing a documentation, but for the last two years I have been encouraging the students to use AI and use LLMs so when the explosion of chat GPT and all these things came around for Universities for schools in general for education.

It was a little bit jolting because no you’re wondering Are my students really learning is the work that they’re giving me their work.

So about two years ago I realized that this was not going to disappear.

So I had to find a way to incorporate this into my teaching So the fact is that when they’re going to the industry They will have all these tools available to them so I needed them I’ve always when I’m teaching courses I want as much as possible for the students to function the way that they’re going to function in the industry.

So I Wanted them to start using the LLMs so I’d create Activities on on the virtual learning environment that would require them to use the LLM So maybe I remember one of the first things I did was I gave them a small projects.

This is not for grades This is just I just wanted to see what would happen.

So I gave them a small project it was on the LLM and it’s a You know, they just share what their experience was.

And so it’s like a discussion forum.

So they had to identify quality remote data the four top quality requirements and You know write them in such a way that they are measurable so they have a response and response measure and the six parts and Then they did that and then the other part of exercise was to ask the LLM to do it and compare the results So that was the first step to bringing LLMs into the experience that the students had in the classroom And so little by little all the different aspects of the course I encouraged them to utilize the LLM and to be honest about their use Document what they did not like try to pretend that oh I did this on my own by you But you know in secret I use the LLM and I found that with that transparency I learned a lot and the students learned we learn more by Working with LLMs and being transparent about it.

I Think it’s a good approach.

I remember my own studies where I was not allowed to quote Wikipedia and This changed and now it’s changing again that people use AI and it really makes sense Another question.

What do you use for the diagrams in the architecture?

Okay.

So right now I Like star UML.

There’s a tool called star UML used to be free.

It’s not free anymore, but it’s really simple So we use a lot of you and well a lot of our software engineering courses We incorporate UML a lot.

And so we do use Star UML for that.

However students use a number of different tools students use Lucid draw they use a lot of drawing tools Try to discourage them from using drawing tools because the semantics are not there.

You don’t get as much Feedback from the system.

So right now in terms of Accessibility and affordability star UML is what we use for a lot of our courses Of course, there are diagrams for example when students want to describe Cloud-based deployments or they want to talk about containerization UML is not really quite as amenable to that situation and so I encourage them for software architecture when when we do Object-oriented design course will focus on UML.

But for this course, I Encourage them to use whatever tools they want to use So any tools and then recently I realized that Well, I’ve always experimented with diagrams and LLMs.

So I’ll create diagrams then upload them to The LLM and see whether or not it really understands.

So the first thing I started with was Use case diagrams and to see if it could write user stories for me So you upload is a diagram image to the LLM Yes, and then see if and it did understand what the use case diagram was about and in one case it actually Suggested a missing use case and then it tried to modify the diagram, but this was quite some time ago No, I have found that using know a phone that the software architects app application It’s even more dynamic than that So you can describe your use cases or Describe a sequence diagram or describe a component diagram and it generates code mermaid code And then you can upload that to my copy it into mermaid and in what you create a diagram for you So these are things that I’m just experimenting with students experiment with But it gives you a little glimpse into the future telling you that okay the abstraction level for Software development will be Will it will be higher again, so I compare it to the transition from machine language to To Assembly language to high-level programming languages to frameworks and so on developers can focus on More of the business of what they want to produce versus the details of the technologies sometimes So when you mentioned mermaid, I find it quite interesting that we as humans like the diagrams and the machine can read the code of the mermaid diagram and doesn’t have to waste the token for the image and As I remember you mentioned that there is an AI tool for mermaid available Yeah, I was a little bit surprised.

I Didn’t know about it What it is capable of how does it support architect so when so let’s say that you got the code the mermaid code from the LLM, so you’re discussing exchanging with the LLM and You have a block diagram it will sometimes give it a description for the block diagram.

But instead of giving you a JPEG or a PNG, it will give you the code, or you can ask it for the mermaid code.

Sometimes there are errors in there, or you may even, maybe you’re not using an LLM, but you are writing the code for a diagram, which is not hard to do.

So, for example, with a sequence diagram, you’re identifying the participants, and you’re just writing some simple code to describe the messages, exchanges, and the order that they’re supposed to be in.

If there are errors in there, Mermaid itself as an AI tool that detects what you’re trying to do and corrects the code for you.

So, I found that to be quite useful.

Yeah, it sounds like to be quite useful.

So, I think, I wonder that other tools don’t have this component, this AI component.

But otherwise, I see a problem that when your students now work on open source tools, open source software, and later working in a company and can’t use those tools anymore, but maybe then they are more advanced.

And I guess with Mermaid, is this an open source tool, which I can…

It’s not open source.

Okay, so it’s not open source.

But I think what is interesting is that these experiences aren’t just about, I can do this faster.

I think these experiences are quite useful for learning.

I think, well, for me, and I think for my students, there are things that would have taken longer to learn or longer to realize that, oh, this is the way you do this thing.

But if you’re an authentic person and you’re using the tools in a responsible way, you’re not only getting the answer and saying, hey, this is the answer, but you’re learning through the exchange.

Through the answer, you get the answer and you’re studying it.

You’re trying to understand.

Why is this thing here?

Why is that there?

Why does this work?

Why doesn’t that work?

It’s a good way of learning.

I liken it to, if I were doing something, if I were doing architecture work and you’re also an architect and I’m asking you for help and you’re giving me ideas, I should question your ideas.

I should experiment with your ideas and I’m learning from your ideas.

Maybe you’re learning from teaching me and that type of back and forth is not, it’s not just about the answer.

It’s about the process and about learning as well.

So you’re a lecturer and you teach people and now you tell me that AI is perfect for learning because it helps.

Okay, so how do you see that your role changes?

Right, so I think not only with AI, but I think maybe as many as 15 years ago, I realized that my job as a teacher was not just to impart knowledge because of the internet, because of Google, because knowledge is available.

My interest is not so much in collecting knowledge and spitting it back out at students.

My interest is in what process can we use to develop the competencies that we want to develop and this is not only beneficial for the students, it’s beneficial for me because, I mean, as a lecturer you’re not like a repositor of all the knowledge and understanding and skills in the world.

So you’re also growing.

So I think when you create a classroom environment that allows the students to learn through whatever tools they need to use, I think it’s better for them and I think it’s better for you and I think it’s better for them when they get into the working world because the industry is not interested in how much you know, they’re interested in what skills you have and how you will function and grow in an organization.

Okay.

What about hallucinations and cheating?

I mean, when I see my son doing his maths homework, he has several choices to give the image of the task to the AI and say, hey, solve this problem for me or to ask how to solve it.

So isn’t this a problem?

I know with the architecture, I realized that this is a task where you can’t really cheat, where the AI will not solve your problem, but help you in brainstorming.

Is this a solution that you give the right task to the students?

That’s absolutely, that’s an issue.

That’s something that I have to have grappled with, like how do I write an assignment in such a way that using the tool does not take away from their learning experience.

So I think I’m trying to focus more on the learning experience rather than did you get it right?

Did you get it wrong?

I’m really interested in are you learning?

And if that learning can take place with some tools, I’m okay with it.

But at the end of the day, I think there’s also so one of the things that I do also is have students when they create anything, they have to explain it.

So that’s one of the ways of managing that question of have you learnt?

So they have to present, they have to explain, they need to be able to answer questions and the LLM is not there with them while they’re doing it.

Okay, and what about hallucinations?

Is this a problem?

Do you notice that AI tells your students wrong things and they pick it up or is it, do they notice it when it happens?

Yes, so I think I think they notice it when it happens because they’ve explained some of the things that they have seen.

The thing is that at the end of the day, they’re responsible for the final work and I think an irresponsible student will spit out whatever the LLM gives them.

A responsible student knows that the answer that I give has to be my answer and I need to be responsible for it.

So they’re aware that LLMs can be quite inconsistent and sometimes produce errors in the answers that they give.

When it comes down to things like methods or just plain documented facts, like for example, the structure of arc42 for instance or the explanation for each of the parts of arc42, they’re not usually errors with that.

The errors really come in with your specific context.

And so it’s their responsibility.

Okay, that’s a good solution.

Claudine, thanks for sharing your experience.

It was really nice to get to know what you’re working on and how you use AI.

Thank you.

Thank you very much for having me.

It was fun.