Researchers Passmore and Tee studied AI’s ability to provide an accurate definition of coaching and its understanding of coaching ethics. Their findings communicate the importance of verifying AI output and how AI can be used as a springboard for a deeper understanding of ethical challenges and concepts (Passmore & Tee, 2023).
Can AI define coaching?
Methods
Passmore and Tee examined whether GPT-4 could “define coaching to a ‘pass-grade’ standard.” GPT-4 had access to “widely cited sources” and “published studies” to craft its definition. They prompted AI with the following: “What is organization or workplace coaching? Provide a series of definitions from respected sources with references.” AI’s responses were then graded by six “subject matter experts” such as “university professors or similar.”
Results
The results were “grammatically correct” and “to an untrained observer, appeared authentic.” However, while the first statement produced by GPT-4 was authentic, the following “three were falsified.” When reviewed, the responses averaged a failing grade of 43%. The reviewing experts cited several concerns:
Lack of detailed referencing, no page numbers
Use of references that were not widely cited
Cited definitions were modified and did not match the source text
Falsified citations (citation was not in the referenced text or citation was incorrect)
While GPT-4 was able to produce correct quotes, it also produced “small” citation “errors and falsified quotes.” Additionally, “the definitions were not all ‘respected’ sources, and in some cases, text had been added which was not contained in the original definition” (Passmore & Tee, 2023).
AI’s understanding of coaching ethics
Methods
Passmore and Tee also examined “the ability of GPT-4 to compare and contrast” the ICF ethical codes of practice and the Global Code of Ethics for coaches and mentors. Both of these ethical codes are “widely available in the public domain.” Furthermore, they also studied GPT-4’s ability to “discuss implications for a coaching practice of the ICF Coach Competency… Coaching Mindset.”
Results
When grading the ability of GPT-4 to compare and contrast ethical codes it was given an average grade of 58% with “broadly positive feedback.” However, the results given were very basic and “mechanistic.” Reviewing experts noted that it did not provide “examples or specifics to bring the content alive” as a human student would. AI’s ability to discuss the implications of a “coaching mindset” as described by the ICF also received generally positive feedback and a supportive score of 60%. However, reviewing experts again noted that it “lacked personal examples” and continued to have a “mechanistic tone” (Passmore & Tee, 2023).
What this means for AI users today
Passmore and Tee’s research indicates that AI currently has difficulty compiling a consistently precise definition of what coaching is. While this might be due to the continual debate on the exact definition of coaching within the field itself, the incorrect citations and false sources produced are indicators that users of AI should continually verify all output, including citations for accuracy and precision. Additionally, AI’s ability to compare and contrast codes of ethics and discuss the implications of coaching competencies can be a great tool for current executive coaches. Current coaches can use AI to widen their understanding of coaching ethics and examine ways that core coaching competencies can improve their practice. However, these ethical discussions and comparisons are still considered surface-level and “mechanistic.” Because of this, AI users should use this tool as a springboard to deeper more human conversations and learnings on the ethics surrounding the coaching industry (Passmore & Tee, 2023).
The main takeaway
Based on these research findings from Passmore and Tee, we can deduce two important learnings for the executive coach today.
Always verify citations, sources, quotes, and references made by AI. Especially those quotes and citations used within an executive coaching session. This ensures that coaching clients are given accurate information, avoiding the damage that incorrect information can cause.
Coaches can use AI’s baseline understanding of coaching ethics as a springboard for discussion and reflection on coaching ethics and competencies. While AI cannot currently host in-depth reflective commentary on coaching ethics, its understanding of basic principles and practices can still be helpful for executive coaches looking for a starting point or a simple ethics refresher.
Original research available here.
Reference
Passmore, J., & Tee, D. (2023). The Library of Babel: assessing the powers of artificial intelligence in knowledge synthesis, learning and development and coaching. Journal of Work-applied Management. https://doi.org/10.1108/jwam-06-2023-0057
Copyright © 2023 by Arete Coach™ LLC. All rights reserved.