by Oana Ignat (Postdoc Researcher, Michigan AI)

Biweekly Workshops for Improving your Confidence and Creativity  in Partnership with VoiCSEs (lunch provided)

Have you ever struggled to speak your thoughts in meetings, share your research ideas with your advisor, or perform in front of your interviewer? You are not alone, many struggle with a lack of confidence, which keeps them from achieving their full potential.

WHY?

During group meetings at and outside the University, I noticed that many people, not just me, were too anxious to participate. Often, few individuals tended to dominate discussions, leaving minority students underrepresented. Despite great efforts to attract a diverse group of students from around the world, many still feel underrepresented in conversations and discussions of viewpoints.

Even at Toastmasters, a club dedicated to improving public speaking skills, approximately 90% of participants were male, and they were already skilled in speaking confidently. The phrase “the rich get richer” came to mind, and I felt a strong urge to act. It is essential to have a space where everyone can participate, regardless of their confidence level, and where everyone receives a gentle nudge to step out of their comfort zone.  In collaboration with VoiCSEs, a student organization that supports and advocates for such efforts, we launched this workshop for us, students, to come together to build on our confidence, understand our inhibitions, and exercise our public speaking.

The motivation for this workshop is twofold: I want to improve my confidence and reduce my anxiety, and I noticed my colleagues could benefit from this as well.

I’ve always focused more on technical skills than “soft skills” such as communication and presenting my ideas, where confidence plays a major role. However, when interviewing for jobs, I noticed my anxiety and perfectionism hindered my performance. As any good researcher, I searched for solutions and found Improv Theater to help me get out of my head and Toastmasters to improve my public speaking skills.

WHAT?

The LaUNCH your CONFIDENCE workshop is a biweekly lunchtime event supported by VoiCSEs, open to all CSE students, both graduate and undergraduate. This workshop is designed to help participants develop their public speaking and confidence skills through hands-on activities focusing on reducing social anxiety, impostor syndrome, and perfectionism while encouraging creativity.

These beginner-friendly activities are adapted from Improv theater and Toastmasters, and aimed to take you out of your comfort zone in an enjoyable way. I typically start the workshop with a round-table game of confidence cards or by watching inspiring videos on relevant topics.

HOW?

> Confidence cards. 

We enjoy using cards from School of Life to break the ice and bring everyone together. Each participant chooses two random cards and shares the one that resonates most with them, explaining why. This ten-minute game encourages meaningful conversation and connection.

> Improv activities. 

Participating in Improv games, such as “Yes, and…” and “Hey waiter!?” can teach one how to listen, be creative, and overcome the fear of making mistakes. These activities are typically done in groups of 2-5 people, who collaborate to create an entertaining and imaginative story using contributions from each participant. The key to success is to listen carefully to your partner’s ideas and build upon them, while also being open to mistakes and not overthinking what you say. Improv promotes a free flow of ideas, and mistakes can even add humor to the game. Check out this video for more on the importance of improv.

> Toastmasters speeches. 

During this activity, participants are given two minutes to deliver an impromptu speech on a random topic, ranging from “What is your favorite fast food” to “What does life mean to you.” The activity promotes quick thinking, improvisation, and public speaking skills. We also learn from watching videos of Toastmasters winners and discussing them, as well as watching TED talks on topics such as improving posture, voice, and calming your nerves. Feedback is provided by one member who will listen carefully to your speech and note things to improve, such as reducing filler words (e.g., um.., like..), using voice intensity, gestures, eye contact, and strategic pauses to engage and connect with the audience.

WHEN & WHERE?

The workshops have been ongoing since the beginning of May and have attracted CSE graduates and undergraduates and succeeded in popularity and involvement. You do not have to have participated before to join us now!

We meet biweekly in the Beyster building during lunchtime (12:00 – 1:00 p.m.). Sign up here for the exact day and room number, as they might vary based on the host’s availability. Please sign up only if you plan to attend, as we will order lunch for attendees.

VoiCSEs is a new CSE graduate student organization that has evolved from ECSEL+. VoiCSEs aims to build a vibrant and inclusive community through community-collaborative events. VoiCSEs supports members who may want to run leadership events but need support with administration, funding, and organization. Stay tuned to learn more about VoiCSEs!

About the author: https://oanaignat.github.io/

by Muhammad Khalifa (PhD student, Michigan AI)

Eight Lessons Learned in Two Years of Ph.D.

Introduction:

If you are starting your Ph.D. soon, you might be wondering about how you should go about it, how to think about your research, which habits to adopt, and which ones to abandon. That was me when I started my program two years ago. Then I started a habit that proved to be quite beneficial: I dedicated a page in my research notebook titled Lessons Learned. I made sure to update this page on a semi-regular basis, typically whenever I learn something from my mentors/advisors, reflect on my progress, or come across an “Aha” moment about something that I should have done differently. This blog post serves to elaborate and expand on some of the key entries on that notebook page. 

As it is almost the end of the summer (sad, I know), I hope this article is timely for the many students who are starting their Ph.D. in the fall. You will find that the first four lessons in this article are high-level, more abstract, and related to the way we should view ourselves as Ph.D. students and our research.

The last four lessons include more practical advice that you can adopt on a day-to-day basis. 

Prior to beginning, I’d like to express gratitude to Prof. Rada Mihalcea for invaluable feedback and to Aurelia Bunescu and Katsumi Ibaraki for proofreading.

Disclaimer: While some of the lessons discussed are related to coding and AI, most of the advice here still applies to those doing their Ph.D. in areas other than computer science.

1. Enjoy the process, not the product

Getting a Ph.D. is a long journey and not only because it is 4-6 years long, but also because it’s hard and this difficulty contributes significantly to its length. In my interview with Wilka Carvalho, a recent Ph.D. graduate from CSE, Wilka described the Ph.D. as the hardest thing he’s ever done. Now I would argue that the difficulty of doing a Ph.D. largely stems from one thing: being so focused on the end product and not enjoying the process of getting there. The end product is usually getting a paper accepted, passing the prelims, or a successful defense. The process is about the long nights spent debugging our code, running an error analysis, and the things we learn along the way. Under this mindset, the Ph.D. is akin to a reinforcement learning environment with extremely sparse rewards (e.g., reward = 1 if paper accepted otherwise 0). This setting is detrimental to our own happiness since the things we do 99.9% of the time are not associated with any reward of any kind. 

Here I would argue that it needn’t be this way and that we can redefine our own research environment to provide plenty of intermediate rewards. Under this new mindset, every code bug we fix is a reward, every draft we write is a reward, every new tool or technique we learn is a reward, etc. This is what Thomas Edison meant when he said “I have not failed. I’ve just found 10,000 ways that won’t work”—Edison defined his own environment where failure is a reward in its own right. Not only did that make him succeed in the end, but also most likely made him enjoy his work. During a Ph.D. (or any research career for that matter) where failure is the norm, we must find satisfaction in the small daily grind. Think about it this way: you are in an environment where you are learning new things every day (and getting paid for it!), surrounded by smart and amazing people, enjoying a great amount of intellectual freedom and autonomy (basically you are your own boss), and you are expected to contribute to the advancement of science. What more can you ask for?  

There is a reason why this is lesson #1: Enjoying the process is the foundation for a happy and productive Ph.D. experience. I will feel like I did a good job if that is the sole takeaway you learn from this article.

2. Understand your research area very well.

One of the defining traits of researchers who excel in their work is their profound understanding of their research area. Rushing to generate ideas and publish papers might seem tempting, but neglecting foundational understanding will impede long-term progress. The lesson here is to immerse oneself in the literature and read extensively about your research area. In the book The Five Elements of Effective Thinking—one of my favorite books and an absolute must-read for researchers—the first foundational element is to understand things deeply. The book argues that understanding something is a spectrum. And more understanding of a given concept can always be achieved.  One example of this is that my co-advisor, Honglak Lee, once sat with many of us for two hours straight going through the derivation of the training objective of diffusion modes. Honglak appreciates the importance of understanding things at the foundational level and this appreciation most likely contributed to his successful career. I have to admit that in my first year of Ph.D., I did not allocate sufficient time to understand my research domain well enough, mostly because I was rushed to come up with a working idea and publish a paper. 

I happened to be one of the students who met with Yang Song while he was giving a faculty candidate talk here at the CSE department. Yang has had a successful Ph.D. by many measures. During the meeting, I asked him to provide one piece of advice for first-year Ph.D. students. His advice was to “Read as many papers as you can in your area and to not worry about publishing in your early years.” At that time it did not make much sense, but now in hindsight, I think this was very valuable advice. Reading a few of the recent papers in your domain is hardly sufficient. What Yang was talking about is reading enough papers to cover most of the literature in your area. Needless to say, it is not only about the paper count you read, although the paper count can serve as a good indicator of how well you are engaged with the literature in your research area. 

While this advice is about building mastery in your own research area, it does not imply you need to know only about your own area. There is another end of the spectrum where students can become too narrowly focused on their own direction that they are unaware of what is being done in other areas. This is something that I would advise against. There is incredible value in knowing about different, but possibly relevant areas. Many great research ideas emerge from the combination of ideas from different domains. Limiting your knowledge and skills to only one research area will likely deprive you of this opportunity.

3. Think in terms of goals, not ideas.

Early in my research career, I used to run after scattered ideas here and there. When you’re starting in a new area without a full understanding of the challenges or limitations, It is very tempting to run after a sole idea that you think will work. The main issue is that ideas have a very short lifespan; an idea is unlikely to work at first, might not be novel enough, might be easily scooped, or might turn out to be inapplicable when you finally sit down to implement it. John Schulman argues that goals have more longevity than ideas. Pursuing a research goal has at least two benefits: firstly, it prevents you from fixating on a single idea and allows you to keep an open mind about different directions that serve your goal; secondly, it reduces the likelihood of getting frustrated because an idea does not work (and trust me, most of your ideas will not work at first) since an idea is only a means to an end.

Let’s assume that you once had an idea to improve the factuality of language models by pairing them with an information retrieval component. You discussed it with your advisor, who approved the direction. You proceeded to implement it, and after two weeks of coding, the idea does not seem to work: in fact, it made the problem worse. If you are thinking in terms of ideas, you’d be easily frustrated and might give the idea a few more attempts before finally giving up and moving on to another, possibly unrelated idea, repeating the same process. However, if your focus was on the higher-level goal of making language models more factual, two scenarios could unfold. You might persist in refining the initial idea while remaining open to trying different techniques, all centered around achieving the ultimate goal. Alternatively, armed with the knowledge gained from working on the first idea, you might move on to a different idea aligned with the same goal, with a higher chance of success. 

On a personal note, I started developing this way of thinking when I began working with Lu Wang, my co-advisor. Whenever I would present some idea to her, Lu would usually map the idea to the high-level end goal behind it. It might take you some time to make this transition in your way of thinking from ideas to goals, but it is certainly a habit you need to cultivate.

4. Value your own research.

With much significant progress happening around us, especially in AI, it has become easy to underestimate the value of our own research. Statistically speaking, only a very small portion of the papers published every year are considered groundbreaking or “game-changing”. This is the nature of scientific research and is an instance of the Pareto Principle, which states that 80% of the change in any area is contributed to by 20% of the population. I’d argue that the tail of the distribution is even longer for AI research. 

I think we all feel this way from time to time. With Large Language Models (LLMs) taking over the field and with many NLP tasks now being considered “solved”, it is easy to question the value of our own research. I felt this way in November 2022 when ChatGPT came out and I am sure I was not alone. The lesson here is despite all these changes happening around us, we should always value our own research. Obviously, this is easier said than done and here I offer you two points that should make this task easier. 

First, realize that each research problem you may work on is important to some group of people, no matter how small this group is. Each paper you put out there will likely be relevant to someone. Even a simple paper describing some experiments, showing some negative results, and providing a discussion on why something does not work is still a valuable contribution to the community. 

When a former Ph.D. student of Richard Feynman, one of the most renowned physicists in history, complained that he was working on problems he did not find “worthwhile”. Feynman’s response was “…The worthwhile problems are the ones you can really solve or help solve, the ones you can really contribute something to.” Feynman goes on to talk about himself: “I have worked on innumerable problems that you would call humble, but which I enjoyed and felt very good about because I sometimes could partially succeed… No problem is too small or too trivial if we can really do something about it.” Moreover, clearly this former student felt bad that he was not famous or a known name in the area, to which Feynman responded: ”You say you are a nameless man. You are not to your wife and to your child. You will not long remain so to your immediate colleagues if you can answer their simple questions when they come into your office. You are not nameless to me. Do not remain nameless to yourself.” 

The second point I want you to consider is that you ought to work on the research you feel is important. While it is important to work in a direction that your advisor supports or one that is “timely” given the current research progress in the community, you also want to work on problems not only because they are easy, but because you feel there is actual value in solving them. I understand there is a risk involved in working on ambitious projects, especially during a Ph.D. when publication matters. In lesson #8, I will discuss a favorite strategy of mine to minimize this risk.

5. Use a Knowledge Management System.

As a Ph.D. student, you are constantly being bombarded with new information. You read papers or articles, you learn a new coding tool, you attend a talk or tutorial, etc. Keeping all this in your head is simply not going to work. Successful retaining of such information can only be achieved via regular reinforcement.  The setup that I have been developing over the last two years comprises mainly two components: A citation management system (CBS) and a research notebook. I personally use Zotero for the former and Obsidian for the latter. Using Zotero or any other CBS is fairly straightforward. I also recommend having a browser extension to be able to quickly add any opened pdf to your library. 

My research notebook consists of two essential parts. A research journal (per project), where I write down on a daily basis what I did on that day and the results I got. I also use it to manage my to-do list for each day. The other part is a bit more involved, which is an idea management system known as the Zettelkasten or the slip-box method. There are multiple articles and books on Zettelkasten so I will not go into detail here. Briefly, the main purpose of the Zettelkasten method is the effective organization of ideas in a way that facilitates idea generation. This method has a bit of a learning curve and requires a level of discipline to commit to. However, I can be very fruitful once it is integrated into your research and thinking process. I would recommend reading a book called How to Take Smart Notes by Sönke Ahrens, which was my main entry to this method.  

6. Know your codebase very well.

In the realm of computational research, code serves as the foundation of experiments and results. In the words of Andrej Karpathy, code is truth. Understanding your code thoroughly is crucial for avoiding unnecessary bugs and ensuring the reproducibility of your results. Nowadays, it is very rare that you will need to implement things from scratch; There are many open-source tools and libraries that make things easier for us. However, that does not necessitate spending less time understanding how these tools work under the hood. 

In my first year, I built my codebase around an existing repository that was not very well-documented. I did not give myself enough time to understand how that code worked, and as a result, debugging and implementing new ideas took up a large amount of my time. This happened because I did not have a high-level understanding of that base repository. The lesson here is to study your implementation thoroughly (even if it’s not your code) to avoid wasting time or, worse, publishing research that is not reproducible.

7. Talk about your work.

Early in my Ph.D., I did not particularly enjoy talking about my work, partially because I felt I was still early in my path and I was worried that my ideas would sound too simple or naive. This was a mistake. You should always talk about your work and even more importantly your work in progress. Discussing your work can bring at least three benefits. Like writing, it forces you to flesh out your ideas more, spot flaws in your thinking process, and refine your ideas. Second, it makes you a better communicator. As you talk to more people about your work, you start to learn how to explain concepts, what examples to cite to support your argument, and what analogies to use. This in turn will reflect on both your writing and presentation skills, making you a better researcher overall. Lastly and as a bonus, the feedback you get may be very helpful. In fact, the idea I got for my first conference paper as a Ph.D. student was during a discussion with a well-known researcher in my area. 

When I mention “talking with people,” I don’t necessarily mean individuals who are experts in your field. In fact, engaging in conversations with people from diverse backgrounds can provide entirely novel perspectives. Inspiration does not abide by rules and it can easily evolve from conversations from people who know nothing about your field.

8. Lead more than one project at a time. 

Now I realize this advice may be a bit controversial. I asked many of my collaborators and my friends in the department about this strategy and I have to say that the majority favored leading only one project at a time. After trying both strategies, I am absolutely in favor of leading more than one project, precisely two. I first encountered this idea when I was talking to Lili Mou, one of the best mentors I ever met. Lili argued that you should pick two projects: one project is ambitious, with a high-risk, high-reward nature, and another project is easier with a “low-hanging fruit” flavor. If your ambitious project did not work out, you can still publish your other project and make progress toward your Ph.D. If it worked out, then you can then publish two projects, one of which is a strong contribution to an important problem. While working on a single project can indeed generate more focus on that single project, the downsides are many. First, it is easy to feel down when you are stuck since that project is all you have going on. Second, you can spend (or rather waste) a lot of time waiting for experiments to finish running because you have nothing else to work on. When working on two projects, if you are stuck on one project, you move to the other; there is always something to do and you do not feel bored at all.

Now I will share with you two tricks that I learned in my second year that will make working on two projects super fun and super easy. The first trick is to work on two projects that are fairly related. This way, you can minimize implementation time by reusing code from one project into the other. That is exactly what I did last year: I was even able to repurpose a full piece of training code in one project for evaluation in the other project. The second trick is one I got from Cal Newport, which is to work on only one project per day. I achieved this by assigning each day in my week to a project while taking meeting times into consideration. This serves to minimize the danger of context-switching between the two projects. Ultimately, we are all different people and what works best for me may not work best for you. However, I recommend you try this for at least a semester and see how well it works for you. 

Conclusion:

This article offered the main insights I gained over the first two years of my Ph.D. I hope you will find them useful, regardless of whether you’re still beginning your journey or preparing to graduate. While I have talked to people who told me that their Ph.D. was a terrible time, I have talked to many others who believe that their Ph.D. years were the best time of their life. Ultimately, I think it is about both your attitude towards the process as mentioned in Lesson #1, and your day-to-day habits. 

Please share your thoughts on the article with me at khalifam@umich.edu. If you have questions or have a different opinion on any of this advice, I would be excited to hear from you!

More about the author: https://mukhal.github.io/

by Ashkan Kazemi (PhD candidate, Michigan AI)

Are you an Internet user worried about the rapid advancement of Artificial Intelligence (AI) and its potential impact on your life and work? You are not alone in feeling this way, as the fear of AI replacing human jobs and decision-making processes is slowly becoming reality. However, Internet users may have more control over AI development than science tabloids often portray.

The release of a powerful new large language model (LLM) technology called ChatGPT in November 2022 has captured public attention, as it is capable of achieving complex intelligence and language tasks previously thought unattainable. ChatGPT has caused concerns about AI wiping out entire sectors and industries. A 2016 Obama Whitehouse report estimated 9% to 47% job loss in the United States due to automation.  While “… jobs that are threatened by automation are highly concentrated among lower-paid, lower-skilled, and less-educated workers,” per the 2016 report’s findings, the recent emergence of ChatGPT and GPT-4 grade AI has caused panic among white-collar workers, whose jobs previously thought to be safe from automation.

In this blog post I aim to paint a brighter future in which we– the users of the Internet, hold the key to shaping the future of AI and how it affects individuals and society through taking control of our data.

Intelligent agents such as ChatGPT, as well as other large language models, are built on top of neural scaling laws. The scaling laws of neural language models dictate that in order for “ChatGPT-like” agents to become more “intelligent”, LLMs require increasing amounts of training data. The data collection process for training large language models involves scraping large amounts of text data from the Internet. This data was then used to train the model, allowing it to generate text and perform tasks such as translation, summarization, and conversation. However, in order for large language models to continue to improve, they will need to learn from even more data. In other words, AI is addicted to the data generated by Internet users. 

In the book Radical Markets, economists Glen Weyl and Eric Posner argue that data should be considered a form of work, since it generates profits for technology corporations in return for “free” services such as search engines and social media platforms. On paper, as a creator of valuable data, you can have agency and control over how your data is collected and used by third party beneficiaries.

Grassroots organizations, such as data co-operatives, have been emerging in recent years to give Internet users the power to reclaim ownership of our data and its benefits. These organizations allow users to sell our data on our own terms, giving us a voice and a stake in the value created from our work. If we are concerned about the regulation of AI and its impact on society, a grassroots approach to data ownership and control may be a long-term solution. By organizing and taking ownership of our data, we can push back and shield aspects of our life from AI, while simultaneously benefiting from our data contributions to AI. Additionally, such frictions on data collection will slow down the development of AI and buy society more time to responsibly develop and deploy it.

If you are interested in more personal and short-term solutions, here are three impactful actions you can take today to protect and control your personal data:

1. Block trackers and advertising bots. Trackers and more broadly the Internet advertising infrastructure keep tabs on our online lives and pool data from different sources to create more accurate models of user behavior. This aggregated data can later be used to train AI models such as ChatGPT. You can stop sharing your data by blocking such trackers. While there are numerous tools and applications that provide this service for free, there is one that stands out: Brave Browser. Brave is a chrome-based browser and supports easy migration options for all popular browsers such as Google Chrome, Firefox or Safari that automatically blocks all trackers and ads. They even pay you for watching Brave’s own advertisements, which you can easily opt out of.
2. Disable tracking across apps on smartphones. You can stop allowing apps on your smartphone to track your activity across applications. This will make it much harder to link and aggregate personal data across different applications, which means less useful free data for AIs to learn from, as well as receiving fewer targeted ads. You can disable tracking on your smartphone by following these instructions.
3. Opting out of data collection on social media and mobile applications. Your Facebook, Instagram, Google, and LinkedIn accounts all track your activity and collect your data. Coincidentally, these companies use user data to train powerful AI models similar to ChatGPT. Fortunately, most of these companies allow you to opt out of data sharing in your account setting. For example, here is how you can opt out of Google’s data collection for your accounts. 

As an Internet user, you have the power to shape the future of AI and its impact on your life. By joining forces with other users and advocating for data ownership and control, you can help ensure that AI is developed in a responsible and ethical manner.

So, let’s unite and make our voices heard!

About the author: https://ashkankzme.github.io

by Muhammad Khalifa (PhD student, Michigan AI)

Introduction: Wilka Carvalho is a 5th year Ph.D. student in Computer Science and Engineering at University of Michigan. His research interests lie at the intersection of deep reinforcement learning, cognitive science, and neuroscience. Wilka is almost at the end of his Ph.D. and so I took the opportunity to ask him questions about his Ph.D., AI research, and advice for aspiring AI researchers.

Muhammad: Hi Wilka, I am super excited to do this interview with you, and thank you for your time. 

Wilka: Hi Muhammad, I am happy to be here. 

Muhammad: Wilka, could you introduce yourself to our readers?

Wilka: Sure, my name is Wilka Carvalho, I am a fifth-year Ph.D. student. I’m co-advised by Profs. Honglak Lee and Satinder Singh here at U-M Computer Science, and by Prof. Richard Lewis in the Psychology department. You will probably see me walking around in between breaks and if you’ve ever talked to me, you’ll learn that I love to talk about how the brain works. I’ll be a neuro-AI research fellow at Harvard after I graduate, where I’ll try to leverage machine learning to advance theories for how the brain works.

Muhammad: You are almost at the end of your Ph.D. journey. How would you describe this process for yourself? 

Wilka: Definitely hard. It is definitely the hardest thing I’ve ever done. Similar to many students here, I used to think I was really good at critical thinking and that my logic was so good. Then I met the standards of my advisors and I learned that my logic was not that good and that my critical thinking had a lot of room for growth. I think the key thing that I’ve learned here is what it means to have thorough or rigorous evidence. 

Muhammad: Could you elaborate on that?

Wilka: I think the basic answer is scale for computer science. Let’s say if an algorithm has property X, it leads to behavior Y. Either I have a [mathematical] proof that says that’s true, or I have to show empirical evidence. For empirical evidence, you want to show it in as many different data sets or environments that test different parts of the algorithm under different conditions. 

Muhammad: I am sure your interests shifted across your Ph.D. How would you describe this transition? 

Wilka: When I began the Ph.D., I was interested in which algorithms and function classes I could use to characterize human cognition and behavior. And for that interest, I have been very influenced by current characterizations of cognition and behavior. For example, we have object-centric representations of the environment. How to get that [in an AI system] in a way that’s general is quite challenging. In the past, I may have been interested in [manually] building this in and seeing what we can do with that. But if you want a really general algorithm, which humans must have, you want to find mechanisms that are very general and that make very few assumptions, and then allow for [object-centric representation] to be like an emergent ability. How to do that is quite hard, and it forces you to think at another level of abstraction.

To answer your question, the shift [over my Ph.D.] has been to be more abstract and to aim for more abstract [methods] that induce what I care about, as opposed to building it in.

Muhammad: How would you describe the ultimate goal of your research? 

Wilka: Generally speaking, I want to understand the brain’s learning algorithms. And at some level, it is going to require working to replicate them.

Muhammad: Many of the readers of the Michigan AI blog are usually either undergrad students or master’s students with an interest in AI or early Ph.D. students. If there is one piece of advice you would give to someone who is considering doing a research career, what advice would you give?

Wilka: It depends on whether you want to go into industry or academia. If you want to go into the industry, then try to get exposure to different topics. So, just work on projects that you think are fun in your Ph.D. or master’s or collaborate in a lab if you are an undergraduate student. Then, you can intern at a company to see what happens in the real world or at least at those companies that you care about.

But if you want to go into academia or a place like DeepMind or Microsoft Research — that’s a bit more academia-like but in the industry — you probably want to develop a vision and a narrative for the kind of research that you want to do. It’s not a commitment, but you want to really go in-depth with a particular topic and persist with it over multiple projects. Soon enough, as you keep working on that particular topic, you start to go beyond the low-hanging fruits and focus on the harder problems.

Muhammad: Looking at the landscape of the current AI research, what are you most excited about at the moment? On the language side, for example, there are Large Language Models like ChatGPT and GPT-4. There have also been some advances in vision like diffusion models and all of that. What excites you the most at the moment?

Wilka: I’m really interested in behavior more than either vision or language. While these recent advances are exciting, I’m not personally excited by them. On another side, I am also concerned that legislation laws are quite slow to enact and I think the rate at which AI is progressing will certainly change the economy faster than the government can act or is willing to act. I feel like we are going to have a big reduction in the workforce.

On another note, I’m personally really interested in successor features. They’re basically this predictive representation of the environment. You represent where you are now as a prediction of where you would go from that place. To give an example, let’s say I’m in the lobby of the Beyster building. I can possibly represent the lobby with a mixture of the auditorium that’s right next door and the CAEN lab and other possible locations. This lets you transfer knowledge across behaviors, and it’s also been implicated in neuroscience a lot. Interestingly, we used to think that dopamine was learning prediction errors for learning to predict rewards, but it actually seems like it’s learning prediction errors for learning successor features instead. 

Muhammad: Is there anything that you don’t particularly like or would change about how AI research is being done at the moment, whether in the industry, big labs, or academia?

Wilka: That’s a good question. There are a few things I don’t like. For instance, I see the importance of benchmarks– you have a standard to compare against, it’s great—but I think then there’s sometimes too large an emphasis on benchmarks. Therefore, they end up being over-optimized and they end up losing their utility, but I don’t know the solution for that. 

Another thing I’m apprehensive about, right now in AI, is that if you are aiming for AI conferences such as ICLR, Neurips, or ICML, you will have a deadline basically every four months or so. I think that this creates an incentive structure where a lot of academic labs encourage students to try to publish frequently at these conferences. And these end up being more often than not smaller projects. And this is not a critique of incremental research. I think incremental research is really important. 

The problem appears when you combine that with the incentive structure of academia, where there’s a large emphasis on things like the h-index and the number of citations you have. This strategy leads us to lose out on bigger, more impactful projects. I think that DeepMind does a good job, where you have these 30, 40-person projects, and they take three years to complete, but then you truly make progress on an important topic. There was a recent article in Nature about how publications generally have a lower impact now or are less groundbreaking, and I would guess, at least for AI, that [our publication culture] is part of the problem.

Muhammad:  What is next for you and your research?

Wilka: I’ve been working on successor features for a little bit now, and it is a nice setup to think about behavior and transferring knowledge. I’ve accepted a position as a postdoc with Dr. Samuel Gershman at Harvard,  in a psychology lab. I’m hoping my future research can connect cognitive theories with successor features. Interestingly enough, the two have not been connected anywhere in literature, not in neuroscience, AI, or in psychology.

Muhammad: What do you plan to do after your postdoc? Do you want to join the industry, or join academia?

Wilka: Right now I’m still pursuing the academic route. I definitely want to look for a healthy department with good culture, good values, and good leadership. I’m not sure yet if it’s going to be psychology or computer science.

Muhammad: Before we wrap up, is there any final advice that you would like to conclude the interview with?

Wilka: Try not to lose sight of what is exciting and enjoyable, and fun about research. Try to work on topics that, again, are exciting and fun.

Muhammad: Thank you Wilka for this interview. We enjoyed having you here.

Wilka: Thank you for having me!

More about Wilka Carvalho: https://cogscikid.com/

by Muhammad Khalifa (PhD student, Michigan AI)

About the author: I am a Ph.D. student in the CSE and I am also affiliated with Michigan AI Lab. My research revolves around reasoning with language models and controllable generation. This blog post will introduce one of my previous research projects — currently in submission. This project was funded by LG AI research and is done in collaboration with Lajanujen Logeswaran (LG AI), Moontae Lee (LG AI and UIC), and my advisors Lu Wang and Honglak Lee.

arXiv link

Introduction 

Answering complex questions is a fundamental aspect of human intelligence. When presented with a question like “What 1988 Christmas comedy film did Brian-Doyle Murray star in?” as humans, we first need to search for movies starring Brian Murray and then identify which of them were released in 1988 during Christmas. Therefore, to answer the previous question we may need to look into the Wikipedia page about Brian Murray to find a list of his movies. Then go through each of them to identify the one released in Christmas 1988 (which is Scrooged). The main challenge is that the facts required to answer such relatively complex questions are often scattered in different locations (web pages, reports, etc.). We need to identify which ones are relevant to answering a given question. 

This task is known as multi-hop question answering (MQA for short), where we are given a question and a large corpus of knowledge and we need to answer the question by performing multiple reasoning jumps (i.e., hops) over the corpus to find the answer. Systems trained for this task are typically comprised of two components: a retriever and a reader. The retriever identifies the relevant documents and the reader produces an answer based on the retrieved documents. Throughout this article, we will refer to the set of documents required to answer a given multi-hop question as the document path. For example, the path to answer the above question is the two-hop path Brian Murray → Scrooged.

Our focus here is on the retriever side, whose training requires examples of questions and ground-truth paths. Unfortunately, it is quite expensive to curate large datasets of question-path pairs, especially for low-resource languages or domains. This creates a bottleneck for building MQA systems. One solution is to resort to heuristics for data labeling. However, this can easily lead to incorrect annotation. For instance, if we want to collect the documents required to answer the question above, we may use all the documents that contain the answer i.e., the word “Scrooged” but that can easily lead us to irrelevant documents. While there are existing data-efficient retrieval methods, they are mainly restricted to single-hop QA. That is, answering questions that require only one document, and it is unclear how to extend them to the multi-hop setting. This calls for data-efficient retrieval methods that can easily extend to the multi-hop setting. Enters our approach, dubbed PromptRank.

Figure 1: An example of PromptRank on the question “What 1988 Christmas comedy film did Bryan Doyle-Murray star in?”. (a) TF-IDF retrieves potential paths and a language model reranker is used to reweigh the paths based on the probability of generating the question. (b) An example path prompt that includes an instruction (shown in red) and the document paths (shown in black). 

PromptRank Overview

PromptRank is inspired by the success of large generative language models trained on massive amounts of data. At its core, PromptRank consists of two steps. First, a simple unsupervised retrieval method picks an initial set of document paths that have the potential to be relevant to the given question. Second, a more complex reranker model reweighs (or reranks) the paths obtained in the first step based on their relevance to the question. The first step is based on a simple unsupervised TF-IDF similarity retrieval. Figure 1. (a) illustrates PromptRank operation on an example question. We will focus on the second step for the remainder of this article. 

Scoring Function

The reranker model is given a path and a question and is expected to output a score that represents how relevant the path is to answering the question. This is achieved by computing the conditional likelihood of the question given the path. More precisely, given a question and a path , we compute the relevance score of the path to the question as:

Where is the conditional probability of generating the question given a prompt according to a pretrained language model. We will later discuss how the path prompt is constructed but let’s first discuss the intuition behind this scoring function.

Basically, the scoring function computes some relevancy score between the question and the document path, which is what we want. However, one might claim that the conditional likelihood of the path given the question is more natural. We argue that is preferred over (for some question prompt ) for two reasons. First, aligns with the pretraining process of LMs, where documents are typically followed by FAQs, questionnaires, and commentaries, all of which include questions relevant to the documents. Second, and more importantly, estimating can be difficult since LMs can be sensitive to surface form [1] making it difficult to compare the likelihoods of different reasons paths using . For instance, shorter paths will tend to have a higher likelihood.  On the other hand, does not suffer from this issue since we compare the likelihoods of the same sequence i.e., the question. 

Path Prompt

Now that we understand the intuition behind the scoring function, let’s dive deep into how it is actually computed. Let’s take the above example. Given the question “What 1988 Christmas comedy film did Brian-Doyle Murray star in?” and the potential two document path consisting of the two documents “ABC” → “Disney”. We first start by constructing a textual prompt that combines an instruction with the content of both documents. The instruction serves as a cue for the language model to nudge it towards assigning higher scores to more relevant paths. One example of an instruction is “Read the previous documents and ask a question”. To obtain performant instructions, we generate a large set of candidate instructions and pick the best-performing ones on a held-out development set. Figure 1 (b) shows how such a prompt would look like for the example we just discussed. 

Instruction Ensembling and Other Techniques

To further boost PromptRank performance, we use a set of techniques that have been proposed to improve prompting methods. For instance, we use instruction ensembling, where a set of instructions are used separately to compute a score for a given path, then such scores are aggregated through max or mean ensembling to obtain the final path score. We also use temperature scaling to scale the output logits when computing the question probability. We also leverage the in-context learning ability of language models by showing the LM examples of questions and their ground-truth paths in context. 

How well does this work? 

To evaluate PromptRank, we compare it to existing fully-supervised systems multi-hop retrieval systems trained on thousands of examples in addition to unsupervised retrieval systems. Figure 2 shows a summary of the performance of PromptRank. Interestingly, with only 128 examples, PromptRank is able to perform closely to fully supervised systems such as PathRetriever [3] and MDR [4] and outperforms DrKit [5]. 

Figure 2: Recall@K of PromptRank compared to unsupervised and fully-supervised systems.

What about the actual QA performance? We also evaluate PromptRank as part of a QA pipeline, where a trained reader model takes as input the paths retrieved by PromptRank and the question and extract the answer span. We compare PromptRank QA performance to fully supervised systems on HotpotQA in terms of the answer exact match (EM) and F1. Table 1 shows the performance, where PromptRank QA performance outperforms or closely matches that of fully-supervised systems. 

Table 1: Full QA performance of PromptRank against unsupervised and fully-supervised systems on HotpotQA test set. ICL refers to in-context learning. refers to the number of in-context demonstrations used. 

Further Analysis

We perform analysis on the effect of the instruction: whether it instruction improves the retrieval performance, and whether automated instruction search provides any benefit. Figure 2 shows performance over different T5 model sizes with three different types of instructions: a single human-written instruction, the best instruction obtained through automated instruction search, and no instruction. Interestingly, using no instruction performs worst, pointing to the importance of having the instruction part in the prompt. Second, automated instruction search seems to yield more performant instructions than manually crafted instructions.

Figure 2: Box plot of R@2 and AR@2 with different kinds of instructions for three different T5 sizes: XL, Large, and Base. The boxplot results are obtained using 200 different instructions.

We perform further analysis on the effect of the instruction location in the prompt, the benefit of joint reasoning over multiple hops, and inference cost using PromptRank. You can read more about that in our preprint, which should be published soon! 

Conclusion

In this post, we have presented PromptRank, a few-shot multi-hop reranking system for open-domain QA. PromptRank is based on language model prompting, where the score of the potential document path is computed as the probability of generating the question given some prompt. PromptRank shows impressive few-shot performance compared to state-of-the-art fully supervised multi-hop retrievers while using as few as 128 examples. We further demonstrated the utility of PromptRank as part of a multi-hop QA pipeline, where combining PromptRank with a reader model achieved comparable QA performance to fully-supervised systems. 

References 

[1] Holtzman, Ari, et al. “Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right.” arXiv preprint arXiv:2104.08315 (2021).

[2] Raffel, Colin, et al. “Exploring the limits of transfer learning with a unified text-to-text transformer.” J. Mach. Learn. Res. 21.140 (2020): 1-67.

[3] Asai, Akari, et al. “Learning to retrieve reasoning paths over Wikipedia graph for question answering.” arXiv preprint arXiv:1911.10470 (2019).

[4] Xiong, Wenhan, et al. “Answering complex open-domain questions with multi-hop dense retrieval.” arXiv preprint arXiv:2009.12756 (2020).

[5] Dhingra, Bhuwan, et al. “Differentiable reasoning over a virtual knowledge base.” arXiv preprint arXiv:2002.10640 (2020).

More about the author

Muhammad Khalifa is a second-year Ph.D. student at Michigan CSE. His research interests involve reasoning with language models, controlled text generation, and reinforcement learning. Muhammad completed his undergraduate studies at Mansoura University, Egypt, and did multiple research internships at Amazon AWS and NAVER Labs Europe. 

by Zhijing Jin and Rada Mihalcea (Michigan AI)

Zhijing Jin recommends: 

How to Avoid a Climate Disaster, by Bill Gates (2021), offers a detailed overview of the current state of climate change and the steps that need to be taken to avoid a climate disaster. The book draws on Gates’ extensive knowledge and experience in technology, business, and philanthropy to provide a unique perspective on the challenges and opportunities presented by climate change. To achieve the goal of getting rid of 51 billion tonnes of greenhouse gases and achieving net carbon zero emissions by 2050, the book discusses the science behind global warming, the impact of greenhouse gas emissions, and the technologies and policies that can help reduce those emissions. It also discusses the role of governments, businesses, and individuals in addressing climate change, and offers practical and innovative solutions for reducing greenhouse gas emissions in various sectors. One of the key messages of the book is that addressing climate change will require a collective effort from all sectors of society and it is important to take action now. Overall, How to Avoid a Climate Disaster is an informative and thought-provoking book for anyone interested in understanding the challenges and opportunities presented by climate change and in finding ways to take action to address this global crisis.

Totto-Chan: The Little Girl at the Window, by Tetsuko Kuroyanagi (1981), is a beautifully written and deeply moving memoir tells the story of Totto-Chan’s childhood growing up in Japan during World War II. Through her experiences at the Tomoe Gakuen, a revolutionary school that placed emphasis on creativity and individuality, Totto-Chan learns valuable lessons about friendship, kindness, and the power of education. This book is a lovely and heartwarming read for anyone interested in education, childhood, and personal growth, and it is full of inspiring tales that will stay with you long after you finish reading it.

Rada Mihalcea recommends:

From among the books I read this year, there are several that I really enjoyed and would recommend. The three that I believe would make a great reading over the winter break:

The Power of US, by Jay Van Davel and Dominic Packer (2021), is a great overview of what we know about the power of groups, and how we can leverage this power to our benefit. As it turns out, group affiliation is one of our strongest psychological traits, and also one that explains much of our human behavior. Consider for example the resolution of religious conflict: the book gives the example of the disagreements among Muslims and Christians in Iraq, addressed through … soccer. All it took to remove the religious tension was to ‘break’ the groups and reconstruct them by assigning people to interfaith soccer teams: the new ‘grouping’ created same-group affiliation among previously disconnected people, and gave their affiliation a new meaning. Or consider the example of misinformation: because of group affiliation, most people would endorse it even when they know it to be fake news just because the author of the misinformation belongs to the same group (think democrats and republicans). The book abounds in such examples and reasons for how and why this works. It also points to new solutions for major world problems, such as climate or poverty, which could be addressed through an “Earth affiliation” (ie, having all humans realize that “we are in this together’’).

Side note: this book was a recommendation from Ben Kuipers.

I Must Betray You, by Ruta Sepetys (2022), is a book that hits close to home. I am Romanian, and I was 15 when the communist regime fell, so I got to live a good part of my life under the darkest years of the Ceausescu dictatorship. This book is a very good documentation of those years, including the informers everywhere; the lack of trust in those around you; the lack of food; the sitting in long lines for absolutely anything (milk, bread, …); the daily energy outage; the cold winters; the communist prisons for the “intellectuals”; the dictator’s portraits in all the classrooms; etc. If you are curious about life under the East European communist block, this historical fiction will give you a sense of how it was like to live during those years. 

Side note: this book was a recommendation from Rada’s daughter, Zara.

Sum: Forty Tales from the Afterlives, by David Eagleman (2009), is a collection of 40 brief (3-4 pg.) stories with imagined afterlives. Each of them is stretching the mind in a different way. In one afterlife, you choose to see how it’s like to be a horse, except that once you change your identity you don’t possess anymore the intelligence that will make you want to become human again; in another afterlife, same-activities are grouped together, so you spend three months doing laundry, then two days tying shoelaces, thirty years sleeping, six days clipping your nails, and so on; in yet another afterlife, everyone is equipped with a camera, to take pictures of everything around and help create a map of the world, except it doesn’t worked as planned, as everyone ends up only taking only pictures of themselves. Fun, creative, and thought-provoking at the same time, these stories will spark your imagination about how an alternate world could work

Side note: this book was found randomly while walking through the AADL library. 

by Ashkan Kazemi (PhD student, Michigan AI)

While fake news detection is an already difficult task for machines (and humans!), what happens when large unmoderated private groups become filled with misinformation? 

Even if there was a perfect algorithm that could automatically detect fake news (which doesn’t exist), it still would require access to the user content to determine its veracity. Although social media platforms usually have access to scan user content to detect malicious purposes, this is simply not the case for end-to-end encrypted social media. 

As a PhD candidate in the AI Lab interested in conducting research at the intersection of NLP and misinformation, I (Ashkan Kazemi) wanted to understand more about the dynamics of fake news on closed messaging platforms.

There is currently no easy way to discover misleading and malicious content on WhatsApp and other end-to-end encrypted platforms at scale, given that all conversations are accessible only to users partaking in the conversations. In a recent paper published in Harvard Kennedy School Journal of Misinformation Review with collaborators at Meedan, MIT and University of Duisburg-Essen, we conducted a case-study of the 2019 Indian General Elections to analyze the usefulness of a crowd-sourced “tipline” through which users can submit content (“tips”) that they want fact-checked. A tipline is an account on social media operated by a fact-checking organization. Social media users are invited to send potential misinformation to the tipline to (a) see if there is a fact-check or (b) share it as a ‘tip’ that the fact-checking organization might investigate.

Using state-of-the-art AI technology for matching similar texts and images, we compared content sent to the election tipline to the content collected from a large-scale crawl of public WhatsApp groups (these are WhatsApp groups where the link to join is shared openly), ShareChat (a popular image sharing platform in India similar to Instagram), and fact-checks published during the same time in order to understand the overlap between these sources.

Our results show the effectiveness of tiplines in content discovery for fact-checking on encrypted platforms. We show that:

• A majority of the viral content spreading on WhatsApp public groups and on ShareChat was shared on the WhatsApp tipline first, which is important as early identification of misinformation is an essential element of an effective fact-checking pipeline given how quickly rumors can spread (Vosoughi et al., 2018).
• The tipline covers a significant portion of popular content: 67% of images and 23% of text messages shared more than 100 times in public WhatsApp groups appeared on the tipline.
• Compared to content from popular fact-checking organizations, the messages sent to tiplines cover a much higher proportion of WhatsApp public group messages. While misinformation often flows between platforms (Resende et al., 2019), this suggests that tiplines can capture unique content within WhatsApp that is not surfaced by fact-checking efforts relying on platforms without end-to-end encryption.

The authors use text and image “embedding models” which are used to map texts and images to mathematical vectors that facilitate processing such complex data at scale. For instance, the authors used PDQ hashing to construct a summary mosaic of about 35,000 images submitted to the tipline. As we move from top left to the bottom right, we see representative images of text turn into image+text combinations across the diagonal, and images in the bottom right are mostly of people. As the visual summary reflects, pictures of newspapers or images with text on them are the most dominant image type submitted to the tiplines under study, constituting over 40% of the content, followed by memes which make up roughly 35% of the content. 

An exciting aspect of this project for me was to develop and use cutting edge technology for non-English language data, as the dataset under study contained text in Hindi, Tamil, Bengali and a number of other Indic languages. Such lower resource languages are often overlooked in the AI and NLP technology development cycles, and working with such data often brings nuanced challenges that make for intriguing research questions. Text and image embedding models were a crucial part of our study, enabling us to take a closer look at the content shared on the Indian election tipline to better understand the gravity of misinformation on private WhatsApp groups. 

Overall, our findings demonstrate tiplines can be an effective privacy-preserving, opt-in solution to identify potentially misleading information for fact-checking on WhatsApp and other end-to-end encrypted platforms.

More about the author: https://ashkankazemi.ir/

(Editor: Jung Min Lee, PhD student, Michigan AI)

by Jung Min Lee (PhD student, Michigan AI)

Dr. Maggie Makar is the Computer Science department’s first Presidential Postdoctoral Fellow/assistant professor. Prior to joining the department in 2021, she was a PhD student at CSAIL MIT, advised by Prof. John Guttag. Dr. Makar’s research interests focus on the intersection of machine learning and causal inference, and how these tools can be applied in healthcare settings. 

The content of this interview has been edited for coherence and readability. 

JM: Maggie, it’s great to see you again. Can you briefly introduce yourself for our readers?

Maggie: Sure, I’m a Presidential Postdoctoral Fellow and a non-tenure track assistant professor here at CSE. Before coming to Michigan, I did my PhD at MIT in CSAIL where I was advised by Prof. John Guttag. Before MIT, I spent a little over a year at Brigham and Women’s hospital studying questions such as the end of life care with Ziad Obermeyer, and before that I was an undergraduate in Math and Economics at the Univ. of Massachusetts Amherst. And before that, I was in Egypt where I was born and raised, and while some people in Egypt will now tell me that my Arabic has deteriorated, I still hold that I am profoundly Egyptian at my core. 

JM: I’m curious about the Presidential Postdoctoral Fellow Program (PPFP), especially since you are the first PPFP fellow in the CSE department. Could you explain a bit more about what that looks like and why you chose to pursue that path? 

Maggie: One of the reasons I ultimately chose to come to Michigan is because of this unique position. As a postdoctoral fellow, I’m somewhere between a postdoc and a full assistant professor. I don’t have any teaching requirements right now, so I can focus my time on research. The position is designed to put you on a fast track to begin your faculty career. Once this appointment is over, I get evaluated on whether or not I will be joining as a tenure track assistant professor. 

I think this position is very helpful because it gives me a bit of space to develop a research agenda, and to think deeply about the type of mentor I want to be and the kind of collaborations I want to build. I like to think of it as a ramp that’s designed to help you start flying. 

JM: Do you plan on teaching once you become a full faculty member? 

Maggie: Absolutely, and that’s something I’m excited about as well. Teaching is a great way to really learn about a topic. There were many times during my PhD when my advisor would ask me “Why is X true? Explain it to me.” And if I couldn’t explain it to him, that told me that I didn’t fully understand the subject. So teaching can really help you understand something better. I also sense some interest from different people in the department for a causal inference course, which is exciting. 

JM: That sounds like a great opportunity. In terms of research, what topics have you been focusing on? 

Maggie: Broadly, the thread of research that I’ve been most excited about in recent years is how we can use ideas from causality to build better machine learning models, and how in turn we can use ideas from statistics and machine learning to make causal inference more efficient. I typically use these tools in the context of healthcare by applying them to specific problems such as infectious diseases. 

I’m also starting to get into causal discovery. It’s a specific type of causal inference where you are not just asking whether or not intervention A has a causal effect on outcome B, but whether you can reconstruct the underlying causal mechanisms based on some observed data. 

Another idea I’m interested in is using causality in machine learning to create more generalizable and robust models. I think it’s important because datasets, especially in healthcare, can reflect systemic biases that are correlations and not actual medical information. If we don’t take into account the causal knowledge of medicine, we might end up with machine learning models that encode spurious correlations or perpetuate biases that have no rooting in medical soundness. 

JM: Do you have any advice for graduate students who are trying to decide between academia and industry careers? What were the factors that made you choose a career path in academia? 

Maggie: That’s a great question. I had offers from both industry and academia so I had to make this choice as well. I think computer science is unique in that you can do great things in either career direction. What ended up tipping the scales for me were two things. First, academia is an exciting career because students tend to bring a fresh perspective that might be different from how I have been thinking about a specific problem. So I like the fact that in academia, there will always be a fresh perspective and that just leads to more exciting research. 

The other reason is that I’m interested in both applications of AI to healthcare and regulation of machine learning algorithms, such as defining what it means to audit them and check if they conform to causal mechanisms. And these are both areas that are harder to tackle in industry. With healthcare, there’s understandably a lot of red tape around medical data that makes it difficult for companies to work with them. There are also some red tape and restrictions around topics such as regulation that might conflict with the companies’ interests, so there is some limitation to academic freedom there. 

That being said, I am still a consulting researcher at Microsoft, so I also enjoy doing industry type research on this side, along with my academic research and my ability to collaborate with students. 

JM: What are some things graduate students can do to prepare for a career after grad school? Were there any specific steps you had to take to prepare for an academic position?

Maggie: Honestly, I believe that what you need to do in order to land a job that you like is roughly very similar independently to whether you want to go into industry or academia. At the end of the day, what companies like Microsoft and Google are looking for is a good researcher. My advisor used to say, “Don’t over optimize. Do what you like doing, and if it gets you where you want to go, then it’s a double win. If it doesn’t go where you want to go, you still have the advantage of having done something you enjoy and having created a dent in an area you wanted to create a dent in.”  

So my advice is to not do things just because it might look good on your resume, or to make the type of calculations like “that looks good if I want to do X and not if I want to do Y”. The reality is that it’s impossible to guess to that level of precision on what you need to do in order to land a very specific position. What you can do is to focus on what you think is worth answering and do a really good job at it. If that gets you the job that you’ve always dreamt of, that’s great. And even if it doesn’t, you still would have done something that’s meaningful to you. 

JM: That’s great advice! I think that way of thinking can also take away some of the pressure students find themselves facing when they become a bit too goal-oriented. It’s also a great segue into our next question. Are there any general advice you have for graduate students? 

Maggie: Yes, I have one piece of advice that I tell everybody. I think one of the things that makes the difference between a successful PhD and an unenjoyable PhD is how you approach failure. I’m definitely guilty of this too –  there are times when I’ve gone into the lab and became anxious or upset that my experiment didn’t give me the result I wanted. The reason this isn’t a helpful attitude is because often, really good papers come out from these failures. If you instead approach this as “That’s really interesting! Let me see why that happened.” you may realize there is an opportunity for a new approach or a new solution. In my experience, good papers are born out of failed experiments most of the time. Instead of just meeting this failure with disappointment and frustration, it is more helpful to let failures spark your curiosity. 

JM: What’s your goal for the next 5 to 10 years, and where do you see yourself?

Maggie: By 5 years, I hope to have built collaborations with students and have them working on good research projects that they are excited about. My hope is that these projects will also have a technical impact and an impact on society, whether it’s through healthcare or positive ethical feedback in AI. 

I hope to be recognized in the causal inference field as someone who’s done good work, and work that moves the field in meaningful ways. And I would hope to be recognized as a good citizen in the department, one who can be trusted by the community in a meaningful way and a person that people feel comfortable reaching out to.

In 10 years – this has been my weird dream – is to be the expert witness in some kind of congressional committee for algorithmic regulation in AI. It’s a fun daydream I’ve had for some time. I’d also like to have something in the hospital system that we’ve implemented, whether it’s an alert system or a diagnostic tool, that can improve hospital operations and patient outcomes. 

More details about Dr. Maggie Makar’s work:

Website: https://mymakar.github.io/

About the author:

Jung Min Lee is currently a PhD student affiliated with the Michigan AI Lab – CSE, EECS Univ. of Michigan.

by Do June Min (PhD student, Michigan AI)

Prof. Laura Burdick is a lecturer in the Computer Science and Engineering division at the University of Michigan. Previously, she was a PhD student at the Language and Information Technologies (LIT) Lab at Univ. of Michigan, advised by Prof. Rada Mihalcea, Director of the AI Laboratory at Univ. of Michigan. Prof. Burdick joined the school as a teaching faculty in 2021 after graduation. She now teaches in the CSE Department at Univ. of Michigan EECS, both freshmen programming courses and courses closer to her research interests in natural language processing (NLP). She also spends time designing courses that are suited for students with limited prior exposure to CS. In this interview, she talks about her journey as a graduate student, researcher, and then as a teacher. The interviewer edited the contents for coherence and readability.

DJ: Laura, nice to see you again. It’s been a while since we last met while we’re both students. Now you’re here again, but in a different role. For the readers who may not know you, can you give a brief introduction of yourself?

Laura: Sure, I am currently a lecturer in Computer Science and Engineering. I started in January, so this is my second semester as a lecturer. Before that, I did my PhD here at Univ. of Michigan advised by Prof. Rada Mihalcea studying NLP. Before that, I did my bachelor’s degree in computer science at Grove City College, which is a small, Christian liberal arts college in Pennsylvania.

DJ: I see, thank you. So currently, your focus lies more heavily on teaching. I remember that while still a graduate student you were passionate about teaching to undergraduates and even to those outside the reach of university programs. For students who are interested/considering teaching, can you share your experience? Specifically, what was your experience with university education like, and why did you choose this path?

Laura: During both semesters that I have been here, I’ve taught ENGR 101, a freshmen programming class that most freshmen engineering students take. I am a part of the teaching faculty. If people are unfamiliar with that career track, there’s generally two faculty career tracks at the University of Michigan: the tenure track faculty and the teaching faculty. For graduate students, your PhD advisor is most likely a tenure track faculty, and they research, teach, and do service to the department and the college. On the other hand, teaching faculty’s primary job responsibility is to teach. Teaching faculty may do a little bit of research, but they are not required to. They typically don’t have PhD students and run research labs like the tenure track faculty. The reason I chose the teaching track was that I really enjoy teaching. I enjoyed teaching as a graduate student, and I enjoy it now, and that was the direction I wanted to go.

DJ: Why particularly at Univ. of Michigan?

Laura: I really enjoyed being at Univ. of Michigan for my graduate work. I already knew a lot of people in the department, So it wasn’t as much of an unknown as going to a new place would have been. Also, I wanted to be in this area for my family. And since I liked the Univ. of Michigan, this seemed like a good fit. I’ve been happy here so far.

DJ: Yeah, you’re not new to Ann Arbor, and it is a pretty amazing place, school-wise and people-wise. You mentioned that your primary mission here is teaching. Can you tell us more about what you are working on? In your case it will be the classes that you’re teaching or other initiatives or projects that you might be involved in as faculty.

Laura: I’ve been involved in teaching and designing several courses. My biggest class has been ENGR 101, a freshman programming class that all freshman engineering students are, for the most part, required to take. We teach MATLAB in the first half of the class, and C++ in the second half, and it’s a huge class! We have over 700 students this semester. With that many students, the class runs like a machine. I’ve been teaching this course for the past two semesters. 

Another course that I will be teaching soon is EECS 492, “Intro to Artificial Intelligence.” It will be my first time teaching that class, and I’m excited to teach that class because the material is related to my research and very interesting. It is a broad course, covering many different areas in AI, and I’m looking forward to learning more about areas that I don’t normally work in.

I’ve also been very involved in a class called “Discover Computer Science (CS),” a freshman level class open to all students but particularly designed for women who have little to no programming experience. One important objective of this course is to help students experience computer science firsthand, by exploring different areas of computer science to see if CS could be a good match for them. I’m not currently teaching this class myself, but my involvement has been to help run the class, develop the class curriculum, and evaluate how the class is working. This course will be running again this winter. I’m excited to keep working on that class.

DJ: I see. So for Discover CS, and for other courses with a specific goal in mind such as, improved engagement from underrepresented groups, it feels like you are doing more than standard teaching work. 

Laura: Yeah, we care about getting more women and underrepresented people involved in computer science. So we try many things, but as engineers, we are actively taking lessons from what we’ve learned in this class and applying them to other places. Moreover, for all classes, there’s an element of scaling up a class and making it better and better each semester, so I think teaching a class is both teaching and curriculum development, which is a lot of fun. It’s interesting to think about how you can best teach computer science so that students are able to learn effectively. Teaching gives you a lot of freedom to try new things in your classroom, and to keep what works and get rid of what doesn’t.

DJ: How easy was the transition from teaching as a graduate student to teaching your job? 

Laura: I was a graduate student instructor (GSI) at the very beginning of graduate school for EECS 281, which is a big class with hundreds of students. So I had some experience in a big class, but I was just a GSI so I taught discussion sections, but I didn’t lecture.

Then I co-taught an NLP class with Rada, my advisor. That gave me more experience lecturing and helping to run a bigger class. I think we had more than 100 students enrolled in that. Then, I was the primary instructor for Discover CS.

My first semester as a lecturer, I taught a class of about 500 students with another lecturer, and it was a bit of a jump! It was really nice to teach this class with someone else for the first time, because I was able to learn a lot from that person.

The other instructor, Laura Alford, was very experienced and has been teaching this class for a really long time, and I learned a lot about how to scale up a class to 700 students, which is a challenge. She showed me how to lead a staff of GSIs and IAs well, and how to organize the class in order to provide the best experience for students.

DJ: Do you have any career advice to students who are interested in a similar career path as you? What can one do as a graduate student?

Laura: If you’re interested in academia, there’s a spectrum of places where you can teach. Typically an academic job is going to have a teaching component, sometimes a research component, and a service component, which is serving on committees and doing service to the department and the college. You have to think about how much teaching versus how much research you want to do. In my job, I do all teaching and I’m not required to do any research, so if I did any research, it would be optional. There are other jobs, for example, a research scientist at the Univ. of Michigan, that are all research and no teaching. And then there are jobs that are kind of in the middle, like the tenure track job at the Univ. of Michigan is half teaching and half research. You can find all these different combinations at different institutions. There are institutions like the Univ. of Michigan, or there are liberal arts institutions where you typically do around 80% teaching, 20% research, depending on the liberal arts institution. In short, if you are interested in teaching and research, there are many types of different institutions that have different balances of teaching and research.

If you decide that you do want to teach, try to get as much teaching experience as you can as a graduate student because that’s going to prepare you well for those job applications and interviews and that position. If you can be a primary instructor for a class, that experience is really going to help you as you look for a teaching job. Talk to your advisor about opportunities to teach, opportunities to GSI, and opportunities to lecture classes. Maybe ask for opportunities to co-lecture the class that your advisor is teaching.

If you’re interested in teaching and are a Univ. of Michigan graduate student, the Engineering Teaching Consultants (ETC) program is an amazing program. As an ECT, you get the chance to  help other GSIs and IAs improve their teaching, and you are also in a community of peers that care about teaching, so you end up learning a lot about pedagogy. You give and receive feedback on teaching, which is really valuable because you see a lot of different teaching styles and a lot of things that work well and things that don’t work well. Plus, it’s a paid program, so I highly recommend it.

DJ: Since we’re running out of time unfortunately, let’s wrap-up with a final question. Where do you see yourself in the next 5 years?

Laura: That’s a good question. While being a teaching faculty here, I am learning so much every semester about teaching. And that’s my goal, I want to continue learning. I want to continue being a better teacher. I want to continue supporting my students better. I want to continue making my classes good places to learn, where people feel welcome and able to learn well.

On a personal note, I love to read, particularly novels, and I’m looking forward to reading many good books over the next few years. I’m currently re-reading Dune, in preparation for seeing the movie soon, which I’ve heard excellent things about.

More details about Prof. Laura Burdick’s work:

Research: https://laura-burdick.github.io/research.html

Outreach: https://laura-burdick.github.io/outreach.html

About the author:

Do June Min is currently a PhD student in the LIT Lab – CSE, EECS Univ. of Michigan.

by Ashkan Kazemi (PhD student – Michigan AI)

NOTE: The original post is live on Meedan’s website: https://meedan.com/blog/claim-matching-global-fact-checks-at-meedan/

The Association for Computational Linguistics conference (ACL) 2021, a top publication venue and event for research in natural language processing (NLP), happened virtually from August 1-6, and I was fortunate to present our own paper at the conference: “Claim Matching Beyond English to Scale Global Fact-Checking”.

From the opening remarks, everyone knew that a major theme of the conference would be large language models and according to program committee co-chair Roberto Navigli, BERT-based models were the most prevalent topic in this year’s proceedings. The rest of the conference came with a fresh and critical perspective, from the opening remarks to discussions around ethics, CO2 emissions and social good in natural language processing (NLP). The conference reminded me of Dr. Strangelove, a cinematic masterpiece by Stanley Kubrick about the threat of nuclear war and how thoughtless actions from a small group of impactful people could endanger humanity. While in the past scientists only speculated about the threat of Artificial Intelligence to humanity, the AI we feared already exists. It was heartwarming and hopeful to see the computational linguistics community engage in these conversations before we hit the point of no-return on harmful AI and language technologies.

ACL Presidential Address

One of the opening talks of the conference was Professor Rada Mihalcea’s presidential address as the 2021 Association of Computational Linguistics president. Coincidentally, she is also my PhD advisor!

Dr. Mihalcea called on the NLP community to “stop chasing accuracy numbers” and expressed that “there is more to natural language processing than state of the art results.” She rightfully pointed out that neural networks have taken over a large part of NLP even though they have major shortcomings such as lack of explainability, concerning biases and large environmental footprints that current NLP benchmarks overlook. The speech followed a year of heated discussions around the ethics and implications of large language models that peaked when Timnit Gebru was fired from and harassed by Google after submitting a paper critical of large language models to the ACM FAccT conference.

Large language models (LLMs) like BERT have transformed natural language processing and played a major role in recent advances in AI technology: powering personal assistants like Siri and Alexa, automating call center services and improving Google search. There are no silver bullets, however, when it comes to these models. A 2019 paper found that training a large Transformer-based language model with neural architecture search emits 5x the CO2 emissions of a car in its lifetime. While a large number of ACL 2021 participants used pretrained LLMs, recently released LLMs like OpenAI’s GPT-3 or Google’s T5 cannot be trained on an academic budget, resulting in a monopoly over this impactful research trend by big tech companies. This is all made worse by the biases encoded in these models including stereotypes and negative attributions to specific groups that make their widespread adoption dangerous to society.

Together with the major limitations of neural networks mentioned in Dr. Mihalcea’s presidential address, we can see that amid the hype surrounding transformer-based language models, they are likely still long away from obtaining human-like intelligence and understanding of language. To quote an anonymous survey respondent from Dr. Mihalcea’s research, “[NLP]’s major negative impact now is putting more money in less hands” and we should be changing our focus from improving accuracy numbers to factors such as interpretability, emissions and ethics to build language technology that benefits everyone, not just a handful of powerful companies.

Green NLP Panel

Green NLP is a community initiative inside ACL that aims to address environmental impacts of NLP. A panel of academic and industry researchers moderated by Dr. Iryna Gurevych discussed these impacts on the second day of the main conference. Dr. Jesse Dodge started the panel with a presentation on “Efficient Natural Language Processing”, an effort to encourage more resource-efficient explorations in NLP and address some of the challenges creative and low budget research face when publication venues are dominated by research around LLMs, without slowing down the impressive progress of NLP seen in recent years. Throughout the panel, many interesting points were brought up by the audience and the panelists.

Time and again, panelists expressed their concern for lack of access to resources required to reproduce a lot of the papers presented at ACL conferences and some even called for interventions. Dr. Mona Diab expressed how instead of the current “Black Box” approach to NLP through the use of LLMs, we should move towards “Green Box”, efficient NLP that is easily reproducible and accessible to a broader and more diverse group of researchers, eventually resulting in democratization of NLP while in parallel reducing emissions of our research. Others pointed out that the current setup in NLP discourages competition from academia and moving towards a more green, efficient NLP could mitigate that and increase creativity in the community’s research output.

The panel ended with a question from the audience, asking how energy use and emissions of NLP compare with cryptocurrencies and Bitcoin. One of the panelists Dr. Emma Strubell elaborated that we simply don’t have an answer to this question yet. While Bitcoin’s energy consumption currently can power up Czech Republic twice, there are active efforts in place to reduce emissions from cryptocurrencies that were simply made possible through measurement, something the NLP and AI community may be lacking behind. There is a lot to be done to ensure NLP is democratized and environmentally safe, but community initiatives like Green NLP spark hope that these hopes could become a reality.

NLP for Social Good

The theme track for the conference, a workshop and a social event shared the same topic: NLP for social good: an effort from Association of Computational Linguistics to nurture discussions around the role of NLP in society. These discussions included efforts to define what “NLP for social good” means, to identify both positive and negative societal impacts of NLP and to find methods for better assessing these effects. Dr. Chris Potts’ last keynote of the conference “Reliable characterizations of NLP systems as a social responsibility” offered detailed and fresh directions for NLP systems that could decrease adverse social impacts of these systems and train and build models that promote performance towards our collective social values.

At the NLP for Social Good birds of a feather social event led by Zhijing Jin, Dr. Rada Mihalcea and Dr. Sam Bowman, a friendly conversation started around questions of community building, current NLP for social good initiatives and directions for the future to develop NLP with positive social impact. The consensus on defining “social good” was to go with a loose and broad definition, as long as we don’t overstate the impact of the research as computer scientists sometimes do. Topics such as NLP for climate change and preserving indigenous languages were brought up as research initiatives that the NLP for Social Good community could focus on in the near future. I unfortunately could not attend the “NLP for Positive Impact” workshop, but I encourage the readers to check out their proceedings.

Ending on some favorite NLP + CSS papers from the conference

• A shoutout for our paper “Claim Matching Beyond English to Scale Global Fact-Checking“in which we try to group similar claims that can be served with one fact-check in a variety of high and low resource Indian languages. We released two novel multilingual datasets for this task.
• Changing the World by Changing the Data
• How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact
• COVID-19 and Misinformation: A Large-Scale Lexical Analysis on Twitter
• Structurizing Misinformation Stories via Rationalizing Fact-Checks
• Tackling Fake News Detection by Interactively Learning Representations using Graph Neural Networks
• Read more about my experience at the 2020 ACL conference